Heterogeneous Multi-Agent Autonomous Learning and Safe Cooperative Decision-Making

Abstract

Unmanned surface and underwater vehicles face challenges in autonomously learning cooperative encirclement for high-value targets under partial observability, intermittent communication, and collision risks. This paper proposes a heterogeneous multi-agent reinforcement learning framework with safe decisionmaking. The framework includes a Dec-POMDP-based heterogeneous architecture that unifies state representation, refines platform attributes, and differentiates action constraints, enabling distinct platforms to learn specialized policies within a shared model. A weighted composite reward function addresses target distance, formation geometry, safety, coordination consistency, action smoothness, and task completion, with an implicit coordination reward embedded to balance local control and sustained cooperation under limited communication. Building on MATD3, this paper introduces Feasible Safe Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (FS-MATD3), which incorporates a dual-safety gateway through training-stage cost and execution-stage correction, alongside a three-stage training strategy (normal, dropout, anomaly) and observation masking to improve robustness against link and node failures. Experiments in a simulated marine environment show that the pursuers' cross-domain success rate converges to a stable level and maintains that stability throughout later training; the evader's maneuverable space shrinks progressively and cross-run variance declines over trials. The method demonstrates consistent reward convergence and task completion reliability across runs, providing effective decision support for safe cooperative encirclement by heterogeneous unmanned platforms.