arXiv cs.AI· Yichen Sun, Qirui Liu, Yan Wang, Jianan Lin, Linbo Jiang, Shiyu Wang, Zhibo Wang, Zhixuan Chu·· 12 小时前AI 评分31
SAVeR²：基于可验证奖励的推理型安全对齐方法，消除大推理模型的“安全税”
SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
AI 导读
SAVeR² 是一种面向大推理模型（LRM）的安全对齐方法，通过四项细粒度可验证奖励信号与难度感知数据选择策略稳定训练，并用保推理梯度投影机制解析安全与推理梯度的冲突。在 1.5B 至 671B 参数规模的模型上，该方法在多个 benchmark 中取得更优安全表现、显著降低过度拒答率，且不损害通用推理能力。
正文 · 原文
Abstract:Large Reasoning Models (LRMs) have achieved impressive performance via explicit Chain-of-Thought (CoT) reasoning, yet this process introduces critical safety risks. We formalize LRM safety objectives as requiring non-malicious reasoning and answers, safety consistency, and explicit intent identification. We argue that existing alignment methods suffer from poor adversarial generalization, a significant ``safety tax'' on reasoning, and unfaithful coarse-grained rewards, primarily due to their neglect of the LRM's intrinsic reasoning capabilities for safety. To address these, we propose \textbf{SAVeR$^2$}, a novel \textbf{S}afety \textbf{A}lignment method via \textbf{R}ewards with \textbf{R}easoning capability. SAVeR$^2$ decomposes the safety objective into four fine-grained verifiable reward signals and utilizes a difficulty-aware data selection strategy to stabilize training. Crucially, we introduce a reasoning-preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating the safety tax. Extensive experiments on models ranging from 1.5B to 671B demonstrate that SAVeR$^2$ achieves superior safety performance and significantly lower over-refusal rates across multiple benchmarks without compromising general reasoning capabilities.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07482 [cs.AI] |
| (or arXiv:2611.07482v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07482 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yichen Sun [view email]
[v1]
Thu, 5 Nov 2026 05:42:45 UTC (1,054 KB)
来源：arXiv cs.AI · arxiv.org