A hierarchical reinforcement learning based subtask-coordinated scheduling method for constrained multi-objective evolutionary algorithm

Abstract

In recent years, constrained multi-objective optimization problems(CMOPs) remain challenging due to the complex structure of feasible regions, the difficulty of balancing convergence and diversity, and the lack of adaptive operator scheduling mechanisms. To address these issues, this paper proposes a hierarchical reinforcement learning–based subtask-coordinated scheduling method for constrained multi-objective evolutionary algorithm (HRL-SCMOE). The proposed framework employs a two-level architecture, where a high-level agent dynamically schedules subtasks–such as forward-oriented exploration, feasibility-driven exploitation, and diversity guidance–according to the environmental state, while a low-level agent adaptively selects variation operators tailored to each subtask. Both agents are trained using Double Deep Q-Networks (Double DQN) and Prioritized Experience Replay (PER) to enhance stability, sample efficiency, and value estimation reliability. Moreover, the algorithm constructs a set of collaborative information pools targeting different search objectives to maintain balanced exploration between feasible and infeasible regions. An adaptive reward mechanism and soft target updates are also incorporated to improve robustness in hierarchical policy learning. Experimental results on three benchmark test suites and four real-world application domains demonstrate that the proposed method consistently outperforms nine state-of-the-art constrained multi-objective evolutionary algorithms (CMOEAs) in terms of convergence, feasibility, and diversity, thereby confirming its effectiveness and strong general applicability.