Risk-Aware Mixed Prioritized Experience Replay for Cooperative Path Planning of UAV Swarms in Complex Dynamic Airspace
TL;DR
Training results show that MPER-MASAC maintains the highest mean reward in the later stage among Multi-Agent Deep Deterministic Policy Gradient, MASAC, and MPER-MASAC, and the effectiveness of the proposed risk-aware mixed prioritized experience replay mechanism in improving task execution performance in the evaluated complex dynamic airspace.
Abstract
: Cooperative path planning for unmanned aerial vehicle (UAV) swarms is a key technique for achieving safe target reaching, dynamic obstacle avoidance, and stable formation maintenance, particularly in complex dynamic airspace. Existing methods often rely on global environmental information or simplified obstacle models, and they are susceptible to interference from low-value experience samples in partially observable scenarios with coupled multiple obstacles, thereby reducing policy learning efficiency and task execution performance. To address these issues, this paper proposes a Multi-Agent Soft Actor-Critic (MASAC) algorithm based on Mixed Prioritized Experience Replay (MPER), termed MPER-MASAC. First, a complex dynamic airspace containing static obstacles, target-region circular moving obstacles, and obstacles in uniform linear motion is constructed, and the cooperative path planning problem for UAV swarms is formulated as a partially observable Markov decision process. Second, under a centralized training and decentralized execution framework, a leader–follower cooperative structure is adopted, and a mixed reward mechanism is designed to jointly guide target reaching, dynamic obstacle avoidance, and formation maintenance. Furthermore, temporal-difference (TD) error and obstacle-related geometric distance variation are jointly incorporated into sample priority calculation, enabling experiences with larger TD errors or stronger geometric risk trends to be replayed preferentially. This mechanism enhances the learning ability of the algorithm in key decision-making scenarios, such as near-obstacle interaction, cooperative obstacle avoidance, and formation adjustment. Training results show that MPER-MASAC maintains the highest mean reward in the later stage among Multi-Agent Deep Deterministic Policy Gradient (MADDPG), MASAC, and MPER-MASAC. In Monte Carlo testing, MPER-MASAC achieves a task success rate of 73.00%, a formation keeping rate of 70.23%, and a mean flight path length of 159 m; the corresponding values are 56.60%, 47.36%, and 192 m for MADDPG, 60.00%, 51.14%, and 185 m for MASAC, and 66.00%, 62.26%, and 165 m for Prioritized Experience Replay-based MASAC (PERMASAC). These results support the effectiveness of the proposed risk-aware mixed prioritized experience replay mechanism in improving task execution performance in the evaluated complex dynamic airspace.