Federated Heterogeneous-Agent Reinforcement Learning for the Flexible Dispatch of Energy Hubs

Abstract

Energy hubs (EHs) are important district multi-energy systems that integrate various carriers for energy conversion and storage. For the flexible dispatch of EHs, reinforcement learning (RL) has been a promising solution, capable of effectively and efficiently handling inherent uncertainties within EHs. However, the sample inefficiency issue of RL could lead to computational burden and suboptimal policies. More recently, federated reinforcement learning (FRL) has emerged as a new paradigm that improves sample efficiency by coordinating the learning processes of multiple agents. In this work, we introduce FRL for flexible dispatch of EHs and, specifically, propose a federated heterogeneous-agent reinforcement learning (FHRL) framework to overcome the challenges posed by inter-agent heterogeneity, which could lead to training instability and performance degradation in FRL. Our framework features an adaptive model aggregation method that extracts generalized knowledge of dispatch policies to generate the global model. Furthermore, it incorporates a knowledge transfer mechanism that leverages the generalized knowledge to guide the optimization of each local policy. Extensive experiments confirm our framework outperforms traditional RL and FRL in dispatch costs while demonstrating scalability in a large-scale scenario involving 50 heterogeneous EHs.