Hierarchical collaborative multi-agent reinforcement learning for hot-rolling production planning with order-splitting flexibility

Abstract

In mass personalized hot rolling, intricate constraints cause load imbalances and low order fulfilment. While order splitting alleviates these bottlenecks, it increases changeover frequency and planning complexity. We propose a bi-level model: the upper level optimizes production cost and time, while the lower minimizes makespan and shutdowns. To address the strong coupling between allocation, splitting and sequencing, a hierarchical cooperative multi-agent reinforcement learning (HCMARL) algorithm is developed using a cross-module closed-loop decoupling mechanism. Decoupled adaptive multi-objective deep deterministic policy gradient (DAM-DDPG) and collaborative constraint optimization Monte Carlo tree search (CCO-MCTS) process continuous allocation and splitting decisions to generate physical constraints. These guide the lower-level flexible coupling production planning optimization (FC-PPO), combined with a corrected generalized advantage estimation (CE-GAE), to synchronize maintenance windows and process adjustments. Experiments show HCMARL closely approaches Gurobi’s exact solutions with significantly higher computational efficiency for industrial-scale problems.