FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA, is proposed and it is proved that FedGSA incurs no additional privacy loss beyond client-side DP training and establishes its convergence under standard assumptions.
Abstract
Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models. However, integrating differential privacy (DP) into federated LoRA remains challenging: independently perturbing and aggregating its two low-rank matrices can cause aggregation mismatch and the quadratic noise term. Existing methods mitigate these issues by freezing one low-rank matrix but still rely on Euclidean aggregation, which is basis-dependent and may distort the global update. To address this limitation, we propose FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA. FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold. In each communication round, clients extract low-dimensional subspaces capturing dominant update directions and encode them as projection matrices. The server aggregates these representations to estimate a geometry-consistent global update subspace and reconstructs the global LoRA factors within it, reducing distortion caused by basis misalignment, privacy noise, and heterogeneous client updates. We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions. Experiments on four GLUE tasks and a language generation benchmark demonstrate consistent improvements across privacy budgets and degrees of data heterogeneity. In particular, FedGSA improves average accuracy over the strongest baseline by 2.17% and 2.27% under $\epsilon=6$ and $\epsilon=3$, respectively.
FraQ, an efficient coordinate-space recompression method for federated LoRA, is proposed, an efficient coordinate-space recompression method for federated LoRA that achieves accuracy close to uncompressed baselines while substantially reducing downlink communication with low server-side recompression overhead.
This work proposes SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator, and introduces a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting.
Yue Xia, Tayyebeh Jahani-Nezhad, Mayank Bakshi et al.· 1 citation
Low-Rank Adaptation (LoRA) has become the standard paradigm for Parameter-Efficient Fine-Tuning (PEFT) within federated learning (FL) under privacy preservation. However, while reparameterization strategies in recent orthogonalization baselines like FedSVD enhance matrix expressiveness, they often overlook the actual dynamic requirements for subspace updates during train-ing. To balance computational efficiency and model expressiveness under dif-ferential privacy (DP) constraints, we propose FedLSA, a Federated Low-rank Subspace Adaptive update framework. Specifically, FedLSA utilizes the cosine similarity of updates to the low rank coefficient matrix B as an economical proxy for subspace drift, which minimizes overhead on the server through dynamic Singular Value Decomposition (SVD) allocation. To complement this mechanism triggered by events, we establish an Output Boundary Calibration (OBC) within the logits space to ensure the optimization trajectory remains robust even when matrix basis updates are suspended. Experimental results on five representative Natural Language Understanding (NLU) benchmarks demonstrate that FedLSA consistently outperforms all evaluated baselines in terms of accuracy. Notably, compared to the strongest state-of-the-art (SOTA) baseline, FedLSA establishes a better Pareto front for efficiency and accuracy.
Zhen-Cheng Fan· Poster Volume 0008 The 2026...· 0 citations
Differential privacy (DP) mechanisms have been widely adopted in federated learning (FL) to enhance model security. However, existing approaches predominantly employ uniform privacy budgets, neglecting personalized requirements arising from heterogeneous user privacy preferences. Such uniform privacy configurations typically necessitate compliance with the most stringent budget, which not only leads to the wasteful underutilization of privacy budgets for certain clients but also compromises overall model utility. To address this limitation, we propose FedSPA, a Subspace Projection Aggregation personalized differential private Federated learning framework. The proposed method conducts singular value decomposition operations on noise-perturbed local models to extract singular value vectors as compact representations of both model structure and privacy noise. The server then clusters clients and identifies a consensus subspace for projecting models with varying noise levels, ultimately aggregating the global model through a residual-aware mechanism. This method not only effectively guides the aggregation of client personalized differential privacy but also reduces communication overhead. Extensive experiments demonstrate the model's effectiveness. Additionally, we provide theoretical proof of the privacy and convergence of FedSPA. Experimental results also showcase its superior performance over personalized DP-FL baselines.
Tianchi Liao, Xiaojun Deng, Lele Fu et al.· Proceedings of the 32nd ACM...· 0 citations
Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.
Lei Wang, Jieming Bian, Letian Zhang et al.· 0 citations
Recent studies have integrated Low-Rank Adaptation (LoRA) into federated learning to enable parameter-efficient fine-tuning of Large Foundation Models (LFMs) across data silos, supporting responsible and privacy-preserving collaboration in distributed environments. However, these studies assume that all clients share the same LFM, so each client fine-tunes LoRA on private data using identical pre-trained weights and the server directly aggregates the resulting LoRA updates for collaborative training. The ideal assumption rarely holds in real-world deployments, where clients typically possess heterogeneous base LFMs. Because LoRA updates are anchored to model-specific pre-trained weights, direct aggregation is not only invalid but often counterproductive, leading to sharply degraded collaborative performance. To address this issue, we propose HeteroFL-LoRA, a federated framework that enables LoRA fine-tuning across heterogeneous LFMs. We express LoRA updates through singular value adjustments, activating task-specific knowledge in an parameter-efficient manner while decoupling them from model-specific pre-trained weights. Then, we introduce a Singular Matrix–Guided Subspace Projection that employs singular matrices to achieve cross-subspace mapping, enabling heterogeneous clients to aggregate their LoRA updates in a unified representation space. In this way, task-specific knowledge can be effectively transferred and shared across different LFMs. Experimental results demonstrate that HeteroFL-LoRA achieves effective and robust collaboration across heterogeneous LFMs while maintaining advantages in both training and communication efficiency.
Zhuojia Wu, Qi Zhang, Xue-Rong Zhao et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
