FedSLM, a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients, is proposed, which provides theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise.
Abstract
Federated learning of foundation models faces a fundamental resource-asymmetry challenge: the institutions holding the most valuable domain-specific data cannot host billion-parameter models. Existing heterogeneous federated approaches attempt to bridge this gap through parameter-efficient tuning, model pruning, or knowledge distillation, yet each trades away a critical property, whether full-model memory reduction, architectural self-containedness, or representational fidelity, leaving the core tension unresolved. We propose FedSLM, a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients. FedSLM uses SVD-based decomposition to produce self-contained client models, whose low-rank subspaces form nested manifolds that are structurally compatible for aggregation. It then applies a two-stage protocol that synchronizes lightweight adapters within compression groups and fuses full-rank reconstructions across groups via structural alignment. Finally, a weak-to-strong elicitation step with auxiliary confidence loss transfers the aggregated knowledge to the full-scale server, while an explicit bias--variance trade-off mitigates compression artifacts. We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise. Experiments on natural language and vision--language benchmarks show that FedSLM outperforms existing federated baselines under both IID and non-IID partitions, while client models operate at roughly 50% of the GPU memory required by the full model.
The heterogeneous nature has been regarded as a predominant challenge during the deployment of federated learning (FL) systems, wherein model heterogeneity—where clients train models of fundamentally different architectures—remains underexplored. Existing methods tolerate it poorly: they enforce interdependent model families, extract sub-models of one shared model, or rely on auxiliary public datasets and proxy models, which constrain model selection or demand storage that resource-limited clients cannot afford. We posit that an effective bridge across heterogeneous models should be both model-agnostic—decoupling knowledge transfer from incompatible parameter spaces—and storage-free, staying practical for clients with sharply different resources. On this basis we propose FedGPA, a hierarchical framework in which clients with comparable resources and identical architectures form a group and a server mediates knowledge transfer across groups of diverse models; at its core, the lightweight, model-agnostic Aligned Co-decision (Alco) Unit aligns class-level prototypical information across groups to bridge heterogeneous architectures, and a prototypical fusion step interpolates prototypes to regularize local training and refine cross-group knowledge. Experiments on CIFAR-10/100, EMNIST, and Tiny-ImageNet across heterogeneous architectures show that FedGPA consistently outperforms strong baselines from four method families, with the largest gains on resource-poor clients, at a cost of only ~100 KB of additional cross-group communication per round. A limitation of this study is that we provide no formal differential-privacy guarantee for the exchanged prototypes and do not address adversarial or data-quality attacks, leaving both to future work.
Liyinglan Liu, Zikai Xiao, Zihan Chen et al.· Journal of Machine Learning...· 0 citations
Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.
Lei Wang, Jieming Bian, Letian Zhang et al.· 0 citations
FEAST is proposed, a federated shared-space training framework that counters this imbalance by jointly training multiple subnetworks within each client's limit by introducing a one-parameter $\gamma$-allocation protocol to control this coupling.
FedTopo is proposed, a relation-level framework that encodes global knowledge as class relation topology, capturing how classes relate within each client rather than where they lie in feature space.
