A heterogeneous compression framework for FedKD is proposed that enables each client to select a compression strategy from a candidate strategy set, and an Adaptive heterogeneouS Compression algorithm for fEderated kNowledge Distillation (ASCEND), which employs an exponential moving average (EMA)-enhanced $\epsilon$-greedy policy to balance exploration and exploitation.
Abstract
Federated learning (FL) enables privacy-preserving distributed model training but faces challenges from heterogeneous model architectures and limited communication resources at the network edge. Federated knowledge distillation (FedKD) alleviates model heterogeneity by combining prototype-wise parameter aggregation and knowledge transfer across heterogeneous models. However, transmitting gradients still introduces considerable communication overhead, while existing compression approaches typically apply a uniform strategy across clients and ignore their diverse model characteristics and resource capacities. To address this issue, we propose a heterogeneous compression framework for FedKD that enables each client to select a compression strategy from a candidate strategy set. We formulate the compression strategy selection problem as a non-stationary stochastic multi-armed bandit (MAB), where each arm corresponds to a compression strategy. An efficiency-aware reward is designed by jointly considering local optimization improvement, global knowledge alignment, and execution time. Based on this formulation, we develop an Adaptive heterogeneouS Compression algorithm for fEderated kNowledge Distillation (ASCEND), which employs an exponential moving average (EMA)-enhanced $\epsilon$-greedy policy to balance exploration and exploitation. Experimental results on multiple datasets demonstrate that ASCEND effectively adapts to heterogeneous model and resource settings, reducing communication overhead and training time while maintaining competitive model accuracy.
FEAST is proposed, a federated shared-space training framework that counters this imbalance by jointly training multiple subnetworks within each client's limit by introducing a one-parameter $\gamma$-allocation protocol to control this coupling.
A domain-aware proxy selection framework to better adopt proxy data for OOD problems is proposed and the experimental results show that the proposed models effectively address the challenges of distribution shifts under OOD with and without proxy data.
Federated learning relies on aggregation schemes that assume all participants train models with identical architectures and a common parameter initialization. While this enables parameter-averaging strategies such as Federated Averaging, it also imposes a strong inductive bias by constraining local models to evolve from nearly identical starting points, potentially reducing model diversity and limiting exploration of the hypothesis space. Recent studies have explored shared-initialization-free and heterogeneous federated learning as largely independent research directions. In this context, we propose a shared-initialization-free, architecture-agnostic aggregation strategy based on meta-learning, where local models generate predictions over a reference dataset that are integrated to train a global meta-model. Among the different forms of heterogeneity in federated learning, this work focuses on statistical and model heterogeneity. The proposed framework is evaluated in both one-shot and multi-round federated settings against parameter-averaging and knowledge-distillation approaches while preserving the independent evolution of local models. Experiments across diverse federated settings, including homogeneous and heterogeneous models, varying numbers of participants, different levels of data imbalance on representative benchmark dataset demonstrate that the proposed strategy in a one-shot scenario achieves competitive performance under limited-data conditions, while iterative refinement of the prediction consensus in the multi-round setting yields consistent improvements over traditional aggregation strategies. These findings demonstrate that prediction-space meta-learning constitutes a practical alternative for federated aggregation without requiring shared parameter initialization. Future work will investigate more advanced consensus mechanisms, adaptive historical consensus strategies, and the evaluation of the proposed framework under larger-scale federated learning environments.
Federated distillation (FD) enables collaborative edge learning by exchanging soft predictions rather than model parameters, offering communication efficiency and architectural flexibility. However, deploying FD over heterogeneous wireless networks requires principled methods to schedule device participation and allocate upload volumes under per-round resource constraints. Existing approaches assume uniform participation or rely on heuristic selection, ignoring the coupling among communication cost, computational capability, and privacy posture across devices. This paper proposes KaaS-Edge, a Knowledge-as-a-Service framework that formulates device scheduling as budgeted submodular maximization. We derive an optimal water-filling volume allocation in closed form and present RADS (Resource-Aware Distillation Scheduling), a greedy algorithm with a constant-factor approximation guarantee. Experiments on CIFAR-100 demonstrate that KaaS-Edge achieves accuracy comparable to full-participation baselines while reducing per-round communication by nearly ten times and cumulative bandwidth by over an order of magnitude, with graceful degradation under stringent privacy constraints.
Sheng-zhi Huang· International Conference on...· 0 citations
Personalized federated learning (PFL) is a promising paradigm for collaborative learning over distributed devices, where edge nodes collaboratively train personalized models without sharing raw data. Although PFL addresses data heterogeneity by learning client-specific models, it still suffers from substantial uplink and downlink communication costs when exchanging high-dimensional parameters in bandwidth-constrained systems. Recent one-bit methods achieve extreme compression, but they usually rely on a single thresholding rule applied to the whole model. This design has two limitations. First, it overlooks layer-wise differences in parameter distributions and quantization sensitivities. Second, a single threshold provides only coarse binary information and cannot capture fine-grained variations in parameter distributions. To address these issues, we propose a communication-efficient PFL framework via layer-wise multi-threshold random sketching. In the proposed method, each layer is assigned its own set of quantization thresholds, so that the compressed representation can adapt to layer-specific statistics while using multiple intervals to provide a finer low-bit description of sketched parameters. The proposed method supports bidirectional communication using compact low-bit sketches and improves the communication-accuracy tradeoff compared with existing one-bit compression approaches.
Xu Zhang, Xing-Yu Hou, Jia-Cheng Cheng et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
