A domain-aware proxy selection framework to better adopt proxy data for OOD problems is proposed and the experimental results show that the proposed models effectively address the challenges of distribution shifts under OOD with and without proxy data.
Abstract
Federated Learning is a distributed machine learning paradigm that trains a global model by aggregating local clients without sharing private data of each client. Federated Distillation (FD) builds upon this paradigm by leveraging knowledge distillation to exchange soft predictions on proxy data instead of model parameters, enabling more efficient communication and supporting heterogeneous model collaboration. However, FD models trained on In-Distribution data are hardly adapted to Out-of-Distribution (OOD) scenarios. In this paper, we propose a domain-aware proxy selection framework to better adopt proxy data for OOD problems. The experimental results show that the proposed models effectively address the challenges of distribution shifts under OOD with and without proxy data by achieving average 82.9\% and 80.6\% over existing works on standard benchmarks. The codes and data are released in https://anonymous.4open.science/r/DPS-FD-8596.
PDKD solves the problem of model drift caused by the inconsistent distribution of distillation datasets and the local data by generating personalized distillation datasets for each client while protecting client data privacy.
Jing-feng Tu, Lei Yang, Chao Ma et al.· ACM Transactions on Knowledg...· 0 citations
Conventional federated learning (FL) relies on parameter averaging, which forces clients to be doubly homogeneous: it demands an identical architecture and degrades under non-IID data. Real-world deployments usually break both assumptions. We sidestep both by building a decentralized knowledge distillation framework in which each client evaluates its peers'model snapshots on its own local data and distills from the resulting soft predictions. Because knowledge is transferred through the shared class posterior, clients are free to run different architectures; and because every teacher is evaluated on the student's own device, raw data never leaves the client, with no central server or public dataset required. Within this setting, we identify and address an under-examined problem: how to combine the peer teacher predictions. Existing methods, like uniform averaging, ignore how knowledge reliability varies across teachers and classes. We propose Class-wise Reliability-Aware Distillation (CRAD), which, per class, first discards teachers that disagree with the peer consensus and then takes a weighted average of the rest, weighting each teacher by its per-class reliability (precision, or inverse variance). Since the variance of an accuracy from $n$ samples scales as $1/n$, support enters automatically: among the teachers that survive filtering, a teacher is trusted for a class to the degree that it is both accurate and well-evidenced for it. On three image-classification benchmarks (CIFAR-10, CIFAR-100, and PathMNIST colon pathology), across heterogeneous architectures under severe non-IID skew, CRAD consistently outperforms competing methods in global accuracy.
Baraa Bilbeisi, Mengchen Fan, Baocheng Geng et al.· 1 citation
This work introduces a novel multi-domain federated learning framework in which lightweight client-side proxy models collaborate with a server-side Foundation Model (FM) to learn new concepts without sharing private data.
Matteo Caligiuri, Francesco Barbato, Pietro Zanuttigh et al.· Trans. Mach. Learn. Res.· 0 citations
Federated learning (FL) is an emerging distributed machine learning framework that enables collaborative learning among multiple parties while preserving data privacy. However, the complexity of environments and node heterogeneity in the real world result in uneven data distribution across nodes, leading to Non-IID (Non-Independent and Identically Distributed) characteristics in data distribution. Such data distribution significantly reduces the convergence and performance of the model, becoming one of the fundamental challenges in federated learning mechanisms. To address the above issue, this paper proposes a novel FL framework-FedGKD. For the Non-IID client data distribution problem, we employ client-side local data augmentation, where GAN models are deployed on each client to generate synthetic samples so that local data distribution imbalance can be effectively alleviated. To further overcome the limitations of client-side local data augmentation under Non-IID, FedGKD introduces server-side privacy-preserving data-free knowledge distillation, which can transfer the knowledge of selected clients to the server while ensuring privacy protection, further mitigating the impact of Non-IID on federated learning and solving the problem of model performance degradation caused by direct aggregation. Extensive experiments demonstrate that FedGKD significantly outperforms the baseline algorithms in terms of accuracy, while exhibiting excellent performance in other metrics.
Furui Zhan, Ziyu Deng, Yingxin Liu et al.· IEEE Transactions on Informa...· 0 citations
Federated Learning (FL) allows decentralized clients to train models collaboratively while preserving data privacy. However, distribution mismatch across clients often leads to poor global generalization and degraded local client-level performance. In such scenarios, some of the clients with their local models trained solely on local data may perform better than the globally learnt model, thus nullifying the benefits of collaborative federated learning. To address this, we propose SAPE-FL (Similarity-Aware Personalized Federated Learning), a novel personalization framework that anchors each client's model to both the global model and a similarity-weighted peer averaged model. By incorporating dynamic, client-specific regularization based on both model similarity and output similarity, SAPE-FL adaptively balances global knowledge transfer and peer collaboration while filtering out dissimilar clients. This dual anchoring mitigates negative transfer and enhances robustness in heterogeneous settings. We theoretically analyze our algorithm establishing its convergence guarantees and empirically show that SAPE-FL outperforms state-of-the-art methods under high statistical heterogeneity and low client data regimes.
A. Kumar, Sunil Gupta, Ngyuen Dang et al.· 0 citations
FedRAM is proposed, a three-step framework that progressively updates two scalar hyperparameters: the task importance weight and the client aggregation coefficient, where the proxy model serves as an intermediate between the local reference model and the global agent model.
Fan Wu, Xinyu Yan, Jiabei Liu et al.· Neural Information Processin...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
