FedTuneFM: Federated Fine-Tuning of Foundation Models for Mobile Edge Computing via Adaptive Compression and Attention Alignment

Abstract

Federated Learning (FL) has emerged as a promising paradigm for fine-tuning large-scale Foundation Models (FMs) in distributed environments while preserving data privacy. However, efficiently adapting FMs in FL remains challenging, especially in mobile edge computing scenarios where devices are resource-constrained and heterogeneous. Existing federated compression–distillation methods often rely on fixed proxy structures and focus mainly on knowledge transfer between the full FM and a single sub-FM. These designs may leave intermediate layers insufficiently updated, accumulate model mismatch, and overlook cross-layer dependencies, thereby weakening knowledge transfer and model performance. To address these limitations, we propose FedTuneFM, an efficient federated fine-tuning framework that jointly optimizes the sub-FM structure, the round-wise update set, and consistency across heterogeneous sub-FMs. FedTuneFM consists of two core modules: the FM-to-sub-FM mapping module and the sub-FM optimization module. The mapping module employs a multi-level compression strategy that selectively retains critical parameters to reduce communication cost. Furthermore, an adaptive block update mechanism selects and updates the most informative blocks in each round. The optimization module aligns attention matrices between the full FM and each sub-FM, as well as across heterogeneous sub-FMs, to improve knowledge transfer and aggregation stability. Experiments on six benchmark datasets, including four text datasets and two vision datasets, demonstrate that FedTuneFM achieves a favorable accuracy–efficiency trade-off in heterogeneous and resource-constrained environments while substantially reducing computational and communication costs.