Multi-Agent Reinforcement Learning Based Latency-Aware Hierarchical Control Enabling Electric-Vehicle Based Virtual Power Plants to Participate in Balancing Markets

Abstract

Balancing markets require flexible resources that can promptly follow dispatch signals. Aggregated fleets of electric vehicles (EVs) operated as electric-vehicle virtual power plants (EV VPPs) are promising candidates. Aggregators must control the total power of EV chargers to track dispatch signals while satisfying individual EV users' charging demands. Conventional centralized optimization methods can achieve high tracking performance. However, they rely on global information and require solving large-scale optimization problems, which impose high computational and communication burdens and limit scalability. To address this issue, this paper proposes a two-level hierarchical control scheme based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. At the upper level, each charging station is modeled as an agent, and at the lower level its policy allocates charging and discharging power to the individual EV chargers. At runtime, each station-level agent uses only local observations and the broadcast dispatch signal. We present a case study on participation in Japan's balancing market Secondary 2 (S2) product. The study evaluates the controller on an EV VPP consisting of five stations with a total of 50 Level 2 chargers. The proposed method achieves a dispatch tracking rate (fraction of dispatch intervals with aggregate power inside the market-defined tracking error band) of 97 percent within the allowable tracking error band around the dispatch signal. It also achieves an 80 percent Target SoC satisfaction rate, where the Target SoC is the user-specified departure-time state of charge (SoC). Overall, this method reduces online computation time and communication latency while maintaining high tracking performance and userdemand satisfaction. These results suggest that MADDPG-based hierarchical control provides a practical control scheme for large EV fleets when latency constraints hinder centralized control.