Alternating Distillation and Resource-Adaptive Pruning for Federated Large Model Adaptation.
TL;DR
This study delves into large PFMs adaptation in the resource-constrained federated learning environment, and proposes an innovative framework, namely ADRAP, which alternates between large model distillation and resource-adaptive pruning, with guaranteed convergence.
Abstract
Adapting large pretrained foundation models (PFMs) to private domains presents significant challenges related to data privacy and limited client resources. In this study, we delve into large PFMs adaptation in the resource-constrained federated learning environment, and propose an innovative framework, which alternates between large model distillation and resource-adaptive pruning, with guaranteed convergence. Our framework, namely ADRAP, confers a dual advantage. First, the alternating pipeline balances retaining intrinsic PFMs capabilities with assimilating new domain knowledge from local private data. Additionally, our novel distilling-while-adaptive-pruning strategy dynamically accommodates heterogeneous client resources, ensuring effective collaborative optimization under arbitrary constraints. Theoretically, we establish upper bounds for parameter updates during both the distillation and pruning-based adaptation stages and further demonstrate that ADRAP maintains an asymptotically optimal convergence rate $O(1/\sqrt{\rho TQ})$. Empirical results reveal that ADRAP outperforms state-of-the-art baselines by up to 14.23% (at $a=0.1$) for NLP tasks and 20.55% (under IID) for CV tasks in average accuracy across both domains, demonstrating its effectiveness and adaptability in federated large model adaptation.