A Multi-Objective Deep Reinforcement Learning Approach for EV Charging Optimization Considering Transformer Lifespan

Abstract

With the large-scale integration of electric vehicles (EVs) into power grids, massive concentrated EV charging accelerates the loss of life (LOL) of distribution transformers. There is a need to reduce transformer LOL while optimizing user experience, thus formulating the EV charge optimization concerning transformer LOL (ECOTL) problem. However, due to its characteristics, such as multi-objective tradeoffs, nonlinearity, uncertainty, and the requirement for time-sensitive online control, the ECOTL problem poses severe challenges in obtaining a high-quality Pareto front, acquiring robust solutions, and improving solving speed. To address these, we propose a multi-objective multi-agent deep reinforcement learning (MOMADRL) method that deploys multiple Critics and Actors for each agent. During training, Critics learn value functions with distinct preferences and guide the corresponding Actors that specialize in learning optimal strategies to update their policies. After sufficient training, each Actor can determine a preference-specific charging strategy in the test phase. Case studies based on real electricity price and load data demonstrate that compared with traditional multi-agent deep reinforcement learning (MADRL) methods, the proposed method achieves better convergence performance on the training set. Moreover, it obtains better Pareto-front approximation than the learning-based baselines while providing fast online inference under the tested simulation conditions.