Skip to content
Preprint

Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning

Aug 2026 · 0 citations · 31 references
Computer Science

TL;DR

TPSP introduces a policy-aware scene encoder to capture the interaction between policy behaviors and surrounding environments, enabling scene perturbation aligned with the current policy, and develops a threat-guided optimization strategy that evaluates perturbed scenes through threat-level differences between policy rollouts on original and perturbed scenes, guiding the generation of safety-critical scenes with higher training value.

Abstract

Reinforcement learning (RL) has shown promising performance in autonomous driving, yet ensuring the safety of online RL policies remains challenging due to insufficient exposure to safety-critical driving scenes. The long-tailed nature of real-world traffic situations makes dangerous and rare interactions difficult to encounter through conventional sampling, limiting the ability of RL policies to learn robust safety behaviors. Existing methods improve training diversity by synthesizing challenging scenes or adversarial situations. However, these approaches typically optimize scene generation objectives separately from the evolving policy, without explicitly modeling how generated perturbations relate to the current policy's weaknesses and learning needs. In this paper, we propose Threat-guided Policy-aware Scene Perturbation (TPSP) for safe autonomous driving with online RL. TPSP introduces a policy-aware scene encoder to capture the interaction between policy behaviors and surrounding environments, enabling scene perturbation aligned with the current policy. Based on this representation, TPSP selectively perturbs critical objects rather than applying uniform modifications across the scene. Furthermore, we develop a threat-guided optimization strategy that evaluates perturbed scenes through threat-level differences between policy rollouts on original and perturbed scenes, guiding the generation of safety-critical scenes with higher training value. Comprehensive experiments demonstrate that TPSP improves safety learning efficiency, achieving strong safety performance on NAVSIM v2 with approximately 4 million kilometers of simulated driving data. Ablation studies verify that policy-aware targeted perturbations provide more informative safety-critical experiences than random or policy-unaware strategies, enabling safer driving under limited interaction budgets.

View source

Similar papers

Preprint Sep 2026

Safety-Critical Scenarios Emerge from Initial Scenes

Safety-critical driving scenario generation has largely focused on manipulating the behavior of surrounding agents while starting from an initial scene from driving data. This assumption can limit the space of discoverable failures, since driving data can provide little opportunity for meaningful interaction. For examp...

Yin Wu, Jia-Rong Wei, Carl Esselborn et al. · 0 citations
Conference Aug 2026

Safe Offline Reinforcement Learning for Autonomous Driving via Causal Risk Features

Offline safe reinforcement learning aims to learn constraint-satisfying policies from pre-collected datasets without online interaction, which is critical for safety-critical applications such as autonomous driving. However, offline datasets collected from heterogeneous sources often contain spurious correlations betwe...

Zi-Qian Wang, Zhen Zhang · 0 citations
#artificial intelligence Preprint Sep 2026

Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning

Safety to Competence (S2C), a two-stage RL framework that separates safety synthesis from competitive task learning, is proposed and it is proved that perfect filtering preserves policy non-exploitability when all players commit to safe maneuvers.

Rui-Han Wu, Rui Yang, Donggeon David Oh et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving

Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offl...

Mahmoud Selim, Cristina Cipriani, K. H. Johansson · 0 citations
Preprint Sep 2026

Safety-aware Skill Adaptation for Reinforcement Learning in Dynamic Environments

Dist-GPRL is presented, a distance-aware and safety-guided reinforcement learning framework for structured robot skill adaptation that sequentially adapts overlapping local windows of sparse trajectory via-points rather than modifying the complete skill at every policy step.

A. K. M. Nadimul Haque, Sheila Sutjipto, Marc G. Carmichael et al. · 0 citations
Open access Aug 2026

Dual-critic constrained deceptive Q-learning for deployment-time policy protection

The Dual-Critic Constrained Deceptive Q-Learning (DCD-Q) method is proposed, a deployment-time trajectory protection framework that aims to reduce the information leaked by released trajectories while preserving acceptable task performance.

Guang-Yu Pan, Bo Hou, Yao Chen et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.