Jul 2026· 2026 ITU Kaleidoscope - AI and Frontier Technologies for Good (ITU K)· pp. 1-8· 0 citations· 21 references
Abstract
The advent of Large Language Models (LLMs) represents a leap toward leveraging AI for the benefit of humanity. However, the realization of this potential requires addressing the growing need for personalization and data privacy across diverse entities. Federated Learning (FL) offers a vital paradigm for developing private and personalized LLMs, yet its effectiveness is often limited by the inequality of computational resources at the edge. Existing federated Low-Rank Adaptation (LoRA) struggles to accommodate such heterogeneous hardware capacities, leading to information loss during model aggregation. To bridge this gap, this paper presents an adaptive aggregation framework designed to optimize federated LLM fine-tuning under these constrained conditions. The framework introduces a dynamic budgeting mechanism that quantifies personalization intensity through Frobenius norm divergence and allocates rank capacity accordingly. To resolve rank heterogeneity, a dimension-aligned strategy based on Singular Value Decomposition (SVD) is applied, enabling the consistent fusion of updates across diverse devices. Extensive experiments on the General Language Understanding Evaluation (GLUE) benchmark using Llama-3-8B achieve an average score of 0.6551 and a mean task ranking of 2.22. The results indicate that our method improves global generalization while preserving client-specific personalization in resource-limited environments.
Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.
Lei Wang, Jieming Bian, Letian Zhang et al.· 0 citations
Local Inference Guided Aggregation for Heterogeneous Training Environments to Yield Enhancement Through Agreement and Regularization (LIGHTYEAR), a federated learning framework that performs update selection in function space using an NTK-based agreement score to characterize predictive behavior and determine a personalized aggregation set for each client.
Mirko Konstantin, S. Zachow, Anirban Mukhopadhyay· 0 citations
A new FAL framework is proposed that utilizes federated representation learning to align client data in a shared embedding space that achieves performance that surpasses existing FAL methods even when they are given substantially larger annotation budgets, demonstrating the value of centralized coordination under privacy constraints.
FraQ, an efficient coordinate-space recompression method for federated LoRA, is proposed, an efficient coordinate-space recompression method for federated LoRA that achieves accuracy close to uncompressed baselines while substantially reducing downlink communication with low server-side recompression overhead.
The Fusion of Layer-wise Exits for Sequential Recommendation (FLEXRec), a discriminative framework that enhances compact LLMs while retaining scalable full-corpus ranking and achieves state-of-the-art accuracy among competing methods while remaining highly efficient.
Xurong Liang, Tong Chen, Q. Nguyen et al.· 0 citations
This survey provides a comprehensive overview of existing research at the intersection of LLMs and FL, and outlines promising future research directions to advance the scalability, efficiency, and ethical deployment of LLMs in federated settings, paving the way for more trustworthy and privacy-preserving natural language processing systems.
Fatiha Ait Baali, Chaima Lhasnaoui, Addi Ait-Mlouk et al.· Machine-mediated learning· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
