Transcending Memory Constraints in Federated Learning via Sequential Block-Wise Training

Abstract

Federated Learning (FL) enables collaborative model training across distributed clients while preserving data privacy. However, the substantial memory footprint required for model training imposes a critical memory wall on resource-constrained edge devices, severely limiting their participation and compromising the inclusiveness of FL systems. To bridge this gap, we propose ScaleFL, a scalable and inclusive FL framework designed to overcome memory bottlenecks via sequential block-wise training. Unlike conventional end-to-end approaches, ScaleFL partitions the global model into discrete blocks and trains them sequentially, thereby drastically minimizing peak memory consumption. Moreover, to address the fundamental challenges of block-wise information loss and inter-block isolation inherent in this paradigm, ScaleFL introduces two synergistic components: (1) a Curriculum Mentor, grounded in information bottleneck theory, which formulates curriculum-aware objectives to guide each block toward structured feature learning; and (2) a Training Harmonizer, which implements a parameter co-adaptation scheme to re-establish bidirectional information flow during both forward and backward propagation. Furthermore, we provide a rigorous theoretical convergence analysis. Extensive empirical evaluations demonstrate that ScaleFL significantly outperforms state-of-the-art methods, achieving up to 84.2% improvement in accuracy, reducing peak memory usage by up to 50.4%, and accelerating convergence by up to 1.9×.