Skip to content

SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards

Nov 2026 · 0 citations
Computer Science

TL;DR

This work formalizes LRM safety objectives as requiring non-malicious reasoning and answers, safety consistency, and explicit intent identification, and introduces a reasoning-preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating the safety tax.

Abstract

Large Reasoning Models (LRMs) have achieved impressive performance via explicit Chain-of-Thought (CoT) reasoning, yet this process introduces critical safety risks. We formalize LRM safety objectives as requiring non-malicious reasoning and answers, safety consistency, and explicit intent identification. We argue that existing alignment methods suffer from poor adversarial generalization, a significant ``safety tax''on reasoning, and unfaithful coarse-grained rewards, primarily due to their neglect of the LRM's intrinsic reasoning capabilities for safety. To address these, we propose \textbf{SAVeR$^2$}, a novel \textbf{S}afety \textbf{A}lignment method via \textbf{R}ewards with \textbf{R}easoning capability. SAVeR$^2$ decomposes the safety objective into four fine-grained verifiable reward signals and utilizes a difficulty-aware data selection strategy to stabilize training. Crucially, we introduce a reasoning-preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating the safety tax. Extensive experiments on models ranging from 1.5B to 671B demonstrate that SAVeR$^2$ achieves superior safety performance and significantly lower over-refusal rates across multiple benchmarks without compromising general reasoning capabilities.

View source

Similar papers

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

This work introduces DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency, and proposes SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers.

Xiang-Yu Zhou, S. Z. Zade, Rafi Ibn Sultan et al. · 0 citations
Open access Oct 2026

Steering Tree-of-Thought Reasoning via Deductive Verification

Large language models (LLMs) have demonstrated great potential in code reasoning tasks, but their reasoning processes lack reliable verification mechanisms, making it difficult to ensure logical correctness. The Tree of Thoughts (ToT) framework improves reasoning by exploring multiple paths and employing backtracking,...

Hao-Liang Cheng, En-Yi Tang, Shuo-Xiao Zhang et al. · 0 citations
#natural language process... Open access Oct 2026

ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMs

As large language models are deployed behind increasingly sophisticated safety mechanisms, evaluating their robustness under realistic adversarial conditions becomes increasingly difficult and critical. Existing jailbreaking evaluations often rely on static prompts, limited interaction horizons, or lightly protected sy...

Md Mushfiqur Rahman, Tobias Rohde, R. Tripathi et al. · 0 citations

ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning

Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable rewards, particularly in mathematics, code, and formal logic. These settings make model accuracy easier to evaluate and optimize, but it remains unclear how far success u...

Ibrahim Ethem Deveci, Funda Tan Çalık, Barış Deniz Sağlam et al. · 0 citations
#natural language process... Preprint Sep 2026

Rewarding Novel Deductions: Solver-guided Process Rewards for Logical Reasoning

S PRING uses SMT solver as a training-time verifier of intermediate reasoning steps to provide process-level supervision and introduces the notion of a novel reasoning step, namely, a step that is logically valid, consistent with the evolving reasoning state, and not already implied by previously accepted non-contradic...

Muhammad Asif Ali, Wen-Qing Wang, Huan Wang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.