Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments

Abstract

Autonomous unmanned aerial vehicles (UAVs) increasingly rely on reinforcement learning (RL) for navigation, yet global navigation satellite system (GNSS) spoofing attacks can induce out-of-distribution observation shifts that corrupt value estimation and degrade mission performance. Existing robust RL methods often lack generalization to unseen attacks. To address this, we propose a curriculum-guided adaptation framework that progressively exposes a robust policy to gradient-based adversarial observation perturbations of increasing intensity while aligning temporal-difference (TD) error distributions across curriculum stages. By preserving TD-error consistency rather than targeting specific attack models, the framework promotes resilience across attack conditions. We further derive a TD-space generalization certificate that bounds performance degradation when the TD-error distribution under a test-time attack remains sufficiently close to that of the final curriculum stage. We evaluate the framework in a UAV deconfliction environment with dynamic 3D obstacles under unseen fixed and dynamic GNSS spoofing attacks. Under fixed spoofing, the curriculum-adapted (CurAd) policy achieves approximately 99-100% mission success across evaluated traffic densities, compared with 20-24% for the best-performing robust baseline, while also outperforming the non-robust vanilla baseline. Under dynamic obstacle-luring spoofing, CurAd achieves the highest episodic rewards and reduces median mission completion steps by up to 22% relative to the best-performing robust RL baseline. These findings demonstrate that adversarial curriculum adaptation transfers resilience from synthetic perturbations to unseen GNSS spoofing attacks without attack-specific retraining, enhancing robust autonomous UAV navigation.