Federated Learning with Consistency Optimization Algorithms under Non-IID Data
TL;DR
This paper proposes Federated Learning with Consistency Optimization Algorithms (FedCO), a novel optimization framework that incorporates a label-skew-aware correction loss and neural feature distribution regularization during local training that significantly improves accuracy and convergence under diverse non-IID settings.
Abstract
: Federated learning (FL) enables collaborative training of deep neural architectures while preserving data privacy, yet its performance often deteriorates in non-IID scenarios, which stems from client-side distribution drift and divergent local updates induced by pervasive data heterogeneity. This challenge is particularly critical for maintaining the structural consistency and generalization of neural models across diverse, distributed sources with significant distribution shifts. In this paper, we investigate how to effectively mitigate label distribution shift and feature distribution skew to enhance the global representation stability of neural architectures. We propose Federated Learning with Consistency Optimization Algorithms (FedCO), a novel optimization framework that incorporates a label-skew-aware correction loss and neural feature distribution regularization during local training. Specifically, our method aligns the internal statistics of architectural components between local and global models to suppress feature-space drift. Combined with an adaptive global aggregation mechanism guided by label entropy, this approach ensures that model updates from heterogeneous clients are consistently integrated into the global architectural parameters. Experimental results on multiple benchmarks demonstrate that FedCO significantly improves accuracy and convergence under diverse non-IID settings. For instance, on CIFAR-10 with extreme heterogeneity ( α = 0.05), FedCO achieves 73.64% accuracy, outperforming FedAvg by 7.32%; it also attains the highest accuracies on CIFAR-100 (62.63%) and TinyIma-geNet (37.82%) under the same setting. Our findings provide a robust strategy for integrating distribution-aware local training with adaptive structural aggregation, offering new insights into enhancing the reliability of distributed neural systems in real-world deployments. The code related to this algorithm is available at https://github.com/Donglin0730/ FedCO.