Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· 0 citations· 25 references
TL;DR
i-FedLoRA provides privacy guarantees, improves model accuracy by up to 3.8%, and expedites training by 1.37-2.23×, and facilitates heterogeneous LoRA aggregation that selectively prioritizes high-confidence knowledge to filter DP-induced noise, thereby achieving robust knowledge transfer.
Abstract
Federated low-rank adaptation (FedLoRA) allows multiple clients to collaboratively fine-tune large language models (LLMs) on downstream tasks without exposing their private data. To mitigate privacy leakage during aggregation, differential privacy (DP) is widely used to clip and perturb local model updates with noise, yet it can compromise model accuracy due to the inherent privacy-utility trade-off. The performance degradation becomes worse under the FedLoRA setting with the amplified DP noise impact and client heterogeneity in both model structure and data distribution. In this work, we propose iP-FedLoRA, a privacy-preserving federated fine-tuning framework for heterogeneous clients that strikes a good privacy-utility balance. Specifically, to fully utilize clients' heterogeneous resources, we customize LoRA modules based on their available resources. iP-FedLoRA employs matrix-wise differentially private local fine-tuning with sensitivity-aware noise allocation and rank-compensated LoRA regularization, which effectively alleviates noise impact of low-rank modules and enhances training efficiency. By leveraging noise-resilient knowledge distillation, iP-FedLoRA facilitates heterogeneous LoRA aggregation that selectively prioritizes high-confidence knowledge to filter DP-induced noise, thereby achieving robust knowledge transfer. Through rigorous privacy analysis and extensive experiments, we show that iP-FedLoRA provides privacy guarantees, improves model accuracy by up to 3.8%, and expedites training by 1.37-2.23×.
Differential privacy (DP) mechanisms have been widely adopted in federated learning (FL) to enhance model security. However, existing approaches predominantly employ uniform privacy budgets, neglecting personalized requirements arising from heterogeneous user privacy preferences. Such uniform privacy configurations typically necessitate compliance with the most stringent budget, which not only leads to the wasteful underutilization of privacy budgets for certain clients but also compromises overall model utility. To address this limitation, we propose FedSPA, a Subspace Projection Aggregation personalized differential private Federated learning framework. The proposed method conducts singular value decomposition operations on noise-perturbed local models to extract singular value vectors as compact representations of both model structure and privacy noise. The server then clusters clients and identifies a consensus subspace for projecting models with varying noise levels, ultimately aggregating the global model through a residual-aware mechanism. This method not only effectively guides the aggregation of client personalized differential privacy but also reduces communication overhead. Extensive experiments demonstrate the model's effectiveness. Additionally, we provide theoretical proof of the privacy and convergence of FedSPA. Experimental results also showcase its superior performance over personalized DP-FL baselines.
Tianchi Liao, Xiaojun Deng, Lele Fu et al.· Proceedings of the 32nd ACM...· 0 citations
Federated Learning (FL) avoids centralizing raw data, but server-side access to per-client updates still creates a significant privacy risk because gradients can leak sensitive information through inversion and related attacks. A common defense is client-level Differential Privacy (DP), which reduces attack fidelity by perturbing individual updates, but often at the cost of degraded predictive utility, especially in heterogeneous settings. In this paper, we study a different design choice: aggregate-only observability, where the server learns only an aggregate update rather than the individual client contributions that compose it. We develop a unified evaluation framework comparing baseline FL, client-level DP-FL, an idealized sum-only aggregation mechanism (hide_sum), and a quantum-inspired aggregate-observation primitive (qsum) based on finite-shot GHZ-style phase estimation. Experiments on Texas PUDF and MIMIC-IV show that the aggregate-only regimes preserve competitive predictive utility relative to baseline FL while removing the direct per-client observation channel used by standard server-side inversion attacks. We also analyze utility sensitivity to shot budgets and the resource scaling of the quantum-inspired primitive. The results suggest that privacy–utility tradeoffs in FL depend not only on how updates are perturbed, but also on what the server is allowed to observe. Aggregate-only observability therefore provides a useful mechanism-level complement to noise-based privacy defenses, while also motivating hybrid designs to address residual leakage from aggregate updates.
Clifford N. Jones, Md Nahid Hasan, S. Wagle et al.· International Conference on...· 0 citations
It is demonstrated that round-wise, layer-wise adaptation can improve the privacy-accuracy-efficiency trade-off of differentially private federated learning.
Wenjing Wei, Alla Jammine, F. Nait-Abdesselam· 0 citations
Split Federated Fine-tuning (SFF) is a promising paradigm for scaling Large Language Models (LLMs) by partitioning model depth between resource-constrained clients and a centralized server. While system incentives for throughput and privacy favor deep partitions, the impact of such configurations on model utility remains poorly understood. In this work, we identify and characterize the Depth-Performance Dilemma: the regime that maximizes system efficiency is precisely where fine-tuning quality collapses. Through a comprehensive audit across four model scales (GPT-2 to Llama-3-8B) and diverse benchmarks, we demonstrate that deeper partitions provide monotonic gains in throughput and privacy at the cost of catastrophic performance plateaus. We evaluate a suite of state-of-the-art federated adapter aggregation methods including AVG, STACK, SVD, and FREEZE, revealing that while these techniques are effective in standard Federated Learning, they fail to mitigate the artifacts unique to split architectures. Finally, we provide a mechanistic diagnosis for this failure, tracing the collapse to the near-isometric topology of Transformers, which allows aggregation noise to propagate without attenuation until it triggers Attention Collapse in the server partition. Our findings challenge the prevailing assumption that partition depth is a utility-neutral tuning knob and provide a structural foundation for stable distributed LLM fine-tuning.
This study experimentally measures the impact of non-IID data, noisy data, and fairness in client selection on model accuracy and convergence, and proposes a privacy-preserving scoring method to assess each client's contribution in FL.
HeteroFL-LoRA is proposed, a federated framework that enables LoRA fine-tuning across heterogeneous LFMs and introduces a Singular Matrix–Guided Subspace Projection that employs singular matrices to achieve cross-subspace mapping, enabling heterogeneous clients to aggregate their LoRA updates in a unified representation space.
Zhuojia Wu, Qi Zhang, Xuerong Zhao et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
