FedKGD: Federated Knowledge-Generating Distillation for Heterogeneous Reinforcement Learning
Abstract
Federated Reinforcement Learning (FedRL) improves sample efficiency while preserving privacy; however, most existing studies assume homogeneous agents or utilize public datasets for knowledge distillation to address agent heterogeneity, which limits its applicability in real-world heterogeneous scenarios. Knowledge distillation (KD) is an effective approach to facilitate knowledge sharing among heterogeneous models, but applying it to FedRL faces challenges such as the scarcity of public datasets and restricted knowledge representation. To address this, we propose a Federated Knowledge-Generating Distillation framework (FedKGD), which autonomously generates pseudo states through a learnable generator, completely eliminating reliance on real public datasets. The generator is optimized with a diversity-maximizing loss to ensure coverage of high-value state spaces; agents achieve efficient knowledge transfer among heterogeneous policy networks by uploading their policy distributions over the pseudo dataset. Theoretical analysis demonstrates that the method converges at a rate of $\mathcal{O}\left( {1/\sqrt T } \right)$. Extensive experiments on various reinforcement learning benchmark tasks show that the proposed framework significantly enhances learning stability and asymptotic performance, and maintains superior results even without relying on any public datasets. This provides a new paradigm for privacy-preserving distributed reinforcement learning.