Aug 2026· Neural Networks· Vol 205 Pt B, pp.
109470
· 0 citations· 42 references
Medicine
TL;DR
This work comprehensively considering the effects of client drift during the training process, and quantifying it as the aggregation error, proposes adaptive gradient regularization, which is based on gradient regularization and further and applies different regularization strengths to each parameter based on the magnitude of the parameter variance between the local model and the global model.
Abstract
Federated learning is a novel distributed machine learning framework with privacy-protection, yet it is vulnerable to the effects of heterogeneous data. Heterogeneous data drive client models that overfit local datasets and depart from the global optimum during local training, which is termed client drift. To address the impact of client drift, we approach this issue from the perspectives of optimization and generalization. We comprehensively considering the effects of client drift during the training process, and quantifying it as the aggregation error. We first propose adaptive gradient regularization, which is based on gradient regularization and further and applies different regularization strengths to each parameter based on the magnitude of the parameter variance between the local model and the global model, thereby mitigating the performance degradation caused by aggregation error and helping model converge to a flatter minimum. In order to obtain the variance between local and global models to compute our adaptive gradient regularization term, we introduce global momentum from the server side as the approximation of global gradient and further utilize it as a gradient correction term. Next, we propose MoFedAGR, which combines gradient correction term and adaptive gradient regularization term, helping client models converge to a consistent flat minimum. We have provided the theoretical convergence bounds of the algorithm we proposed. Furthermore, experiments on several image classification datasets demonstrate that our algorithm significantly improves model performance while exhibiting strong generalization capabilities.
This paper proposes Federated Learning with Consistency Optimization Algorithms (FedCO), a novel optimization framework that incorporates a label-skew-aware correction loss and neural feature distribution regularization during local training that significantly improves accuracy and convergence under diverse non-IID settings.
Ruiqi Wu, Yehong Li, Hongjie Guo et al.· Computers, Materials & C...· 0 citations
Local Inference Guided Aggregation for Heterogeneous Training Environments to Yield Enhancement Through Agreement and Regularization (LIGHTYEAR), a federated learning framework that performs update selection in function space using an NTK-based agreement score to characterize predictive behavior and determine a personalized aggregation set for each client.
Mirko Konstantin, S. Zachow, Anirban Mukhopadhyay· 0 citations
Asynchronous federated learning improves the efficiency of conventional synchronous protocols by integrating updates as they arrive. However, asynchrony and data heterogeneity make learning objectives at global and local levels inherently inconsistent—global optimization trajectories can conflict with ongoing local updates. Existing methods simply distribute the latest global weights to clients, which may overwrite local progress and cause model drift. We propose OrthoFL, an orthogonal calibration framework that decouples global and local learning progress to reduce interference. Upon receiving an update, the server aggregates it into the global weights via an adaptive moving average. For local training, OrthoFL computes the global shift due to updates from other clients during the client's delay and removes its projection onto the received update. The resulting parameters lie in a subspace orthogonal to the client update and preserve the closest component of global progress within the orthogonal hyperplane. The calibrated shift is then merged into the client model for further training. Extensive experiments across five datasets demonstrate that OrthoFL improves accuracy by 7.4 percentage points and achieves up to a 12× speedup over the best-performing synchronous baseline. Moreover, it consistently outperforms state-of-the-art asynchronous baselines under various delay patterns and heterogeneity scenarios.
Jiayun Zhang, Shuheng Li, Haiyu Huang et al.· Proceedings of the 32nd ACM...· 0 citations
This paper proposes a robust decentralized personalized federated learning method R-DPFL, that enables clients to reduce the impact of Byzantine attacks via robust neighborhood direction estimation and history-based update trend prediction, rather than purely aggregating client models as in the existing work. In R-DPFL, each client first computes the current-round model update by aggregating the received neighborhood update vectors. It then predicts what this update should be based on its historical values and local model changes. Finally, R-DPFL computes the difference between these two quantities, adaptively clips this difference, and adds it to the local update. We prove convergence of the learning process through rigorous analysis and show that honest clients maintain stable personalized descent dynamics under Byzantine neighbor perturbations without requiring consensus among neighboring models. Extensive experiments on CIFAR-10 demonstrate that RDPFL consistently outperforms state-of-the-art decentralized and personalized federated learning baselines under heterogeneous and adversarial settings.
This work proposes RAVEL-FCL, a generative replay-based framework for federated continual learning that integrates an improved generative model based on Rebooting ACGAN with multi-level feature alignment to ensure consistency and employs Elastic Variational Continual Learning on the server to probabilistically regularize the global model and preserve past knowledge.
Yurui Zhou, Jia Hu, G. Min et al.· ACM Transactions on Autonomo...· 0 citations
Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner, and proves that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error.
