Hierarchical PPO–TD3 Energy Management for Renewable-Rich Microgrids with Safety-Filtered Action Projection

Abstract

For renewable-rich microgrids, effective energy management is essential to improve economic efficiency and renewable-energy utilization while maintaining supply adequacy and network security. However, online dispatch remains challenging because variable renewable generation and demand must be coordinated under battery operating limits, distribution-network voltage constraints, and limited forecast information. To address this challenge, this paper proposes a safety-filtered hierarchical PPO–TD3 framework for microgrid energy management. The upper PPO layer selects coarse battery operating modes for long-horizon economic scheduling, while the lower TD3 layer refines continuous battery-power commands for short-term regulation. Before execution, the fused command is projected through a model-based safety filter considering state-of-charge limits, supply-priority rules, and voltage screening under a 33-bus radial-network surrogate. Fixed-test evaluation over five random seeds and 62 chronologically held-out test days shows that the proposed controller reduces total operating cost from 1003.66 to 686.31 k c.u. (31.6%) and unserved energy from 1743.25 to 1052.14 kWh (39.6%) relative to MPC-Lite, while reducing average decision time from 180.21 to 43.55 ms per step. In a separate safety-filter ablation, the filter reduced total cost from 912.47 to 686.31 k c.u., the hard-voltage violation rate from 25.31% to 19.46%, and the executed state-of-charge violation rate from 0.40% to 0.00%. These results indicate that the proposed framework supports reliable and efficient renewable-rich microgrid operation, facilitating renewable-energy integration and enhancing operational resilience.