Formation control for multiple similar autonomous underwater vehicles with dynamic gaps via deep reinforcement learning

Abstract

Dynamic gaps between simulation and real world environments are the major challenge for the implementation of deep reinforcement learning (RL) based formation control methods in systems composed of multiple similar autonomous underwater vehicles (AUVs), mainly involving hydrodynamic parameters and unknown disturbances. For this reason, this work decomposes the formation control of multiple similar AUVs into the target point generation problem at the kinematic level and the tracking control problem at the dynamic level. At the kinematic level, a desired position generator is designed to generate target points to be tracked for each AUV in the formation. At the dynamic level, the state space, action space, multi-objective reward function and dynamic gap randomization components are customized for the target point tracking control task of the AUV. It is worth mentioning that the dynamic gap randomization components constructed based on prior knowledge provide robustness to the gaps for the learned policy, which enables the policy learned in simulation to be directly transferred to a real environment. Meanwhile, the designed velocity subspace and gap subspace further enhance the robustness of the learned policy. These subspaces alleviate the loss of policy optimality caused by randomization operations. Finally, by integrating the above components, a deep RL based hierarchical formation control method for multiple similar AUVs with dynamic gaps is proposed, and the effectiveness of the proposed method is verified through ablation simulation experiments.