Personalized Federated Learning via Double Knowledge Distillation for Heterogeneous Model Deployments
Abstract
Due to distributed data and privacy concerns, federated learning(FL) is a promising approach to learn global models from distributed data, with personalized federated learning (PFL) being a key enabler for customized services in future 6G networks. Federated Distillation (FD) is a classic communication-efficient PFL paradigm. It uses knowledge distillation to learn the strategy of model personalization. Although the original FD algorithm, also known as FedDistill, theoretically supports the direct deployment of heterogeneous model structures across clients, it can degrade the accuracy performance. To address this, we propose a new PFL method, Personalized Federated Double Knowledge Distillation (pFedDKD), that exploits knowledge distillation in both the global aggregation and local update phases. Moreover, we develop a variant, pFedDKD-SSBLA, that adaptively aggregates the highest-quality logits generated across different model depths. The experiment shows that the proposed pFedDKD and pFedDKD-SSBLA not only show better performance on average accuracy than FedDistill with heterogeneous models deployed, but also consistently outperform the state-of-the-art (SOTA) baseline, FedProto.