Federated Learning in the Era of Large Models: Core Challenges, Representative Solutions, and Emerging Directions

Abstract

With the rapid advancement of the Internet and the emergence and popularity of large language models, the demands for high-quality data, communication capacity and security and privacy have increased significantly. However, a large amount of data is distributed across different devices, and cannot be collected legally because of security and privacy constraints. Thus, Federated Learning was proposed to address these issues. Federated Learning enables multiple clients to train model collaboratively without uploading raw data. Federated Learning serves as a key role in tackling data problems in the era of large language models. According to this condition, it is important to understand the state-of-the-art of Federated Learning. Using literature review and comparative analysis, this paper examines three core challenges in Federated Learning: heterogeneity, communication overhead and security and privacy risks. It also reviews representative solutions based on drift correction, meta-learning and homomorphic encryption, etc., to these three problems. The analysis shows that these methods generally improve one or two objectives. Heterogeneity alleviation may increase model complexity, and privacy mechanisms may introduce additional computation and communication costs. Moreover, large language models further intensify these trade-offs. Parameter-efficient fine-tuning improves the possibility of federated large model training via freezing the backbone weights and transferring a small subset of parameters. Consequently, Federated Learning is moving from full-parameter model collaboration to lightweight knowledge collaboration, with potential advancement toward multi-agent assignment coordination.