Safe Reinforcement Learning With Dual Robustness via Robust Control Barrier Functions
Abstract
In this paper, we propose a novel framework for dual-robust safe reinforcement learning via robust control barrier functions. This framework eliminates the gradient conflicts caused by the inconsistency between the adversarial attack terms in the performance and safety objectives, thereby enabling the use of the Lagrange multiplier method for robust safe reinforcement learning. The key idea is to reformulate the robust safety constraint using a robust discrete-time control barrier function (RDTCBF), which replaces the safety-related adversarial term with an adversary-independent worst-case safety margin by leveraging the Lipschitz continuity of the RDTCBF. We then analyze the conservatism induced by the Lipschitz relaxation, and further develop a local Lipschitz bound, leading to a less conservative safety constraint while preserving robustness Finally, we employ a state-dependent Lagrange multiplier to transform the constrained problem into an unconstrained minimax formulation, and theoretically show that the state-wise safety constraints are not relaxed into a global average constraint. Experiments on safety-critical benchmarks demonstrate that the proposed method improves performance by an average of 10.40% over state-of-the-art baselines while maintaining robust safety. Real-world quadrotor experiments further validate its effectiveness. Note to Practitioners—Safe reinforcement learning provides an effective approach to safety-critical control problems. However, policies trained in simulation may experience performance degradation and safety violations when deployed in real-world systems due to model errors and external disturbances. To address this challenge, this paper presents a robust control barrier function-based safe reinforcement learning method that enhances the robustness of both task performance and safety within a unified adversarial learning framework, thereby facilitating practical deployment in autonomous systems.