arXiv cs.AI· Jiawei Chen, Yi Zhang, Jiawei Du, Zhonghou Lv, Tairan Chen, Junyuan Shang, Jinchang Hou, Xiao Yang, Yu Tian, Chenfu Bao, Chao Yu, Zhaoxia Yin·· 12 小时前AI 评分34
重新思考 GRPO：面向大型推理模型安全对齐的 Beacon 框架
Rethinking GRPO for Large Reasoning Model Safety Alignment
AI 导读
针对 GRPO 在大型推理模型安全对齐中的不足，研究者提出 Beacon 框架，包含安全信号恢复（SSR）与安全-效用解耦对齐（SUDA）两部分，用于解决优势坍缩和不安全多样化两种失效模式。在四个 Qwen3 与 DeepSeek-R1-Distill 变体、八个基准上，该框架取得最佳平均安全与推理表现，平均安全错误率最多降低 32.83 个百分点，同时保持较低的过度拒答。
正文 · 原文
Authors:Jiawei Chen, Yi Zhang, Jiawei Du, Zhonghou Lv, Tairan Chen, Junyuan Shang, Jinchang Hou, Xiao Yang, Yu Tian, Chenfu Bao, Chao Yu, Zhaoxia Yin
Abstract:Large reasoning models (LRMs) exhibit strong reasoning capabilities but remain vulnerable to unsafe outputs. While Group Relative Policy Optimization (GRPO) is widely used for safety alignment, its group-relative optimization may provide insufficient safety-directed learning signals. In this paper, we identify two failure modes: advantage collapse, where homogeneous rollout groups yield vanishing advantages, and unsafe diversification, where unsafe behaviors persist and become increasingly diverse. Our theoretical analysis reveals that behavioral consistency weakens within-group safety contrasts, while entropy regularization can diversify unsafe responses when relative signals vanish. To address these challenges, we propose Beacon, comprising Safety Signal Recovery (SSR) and Safety-Utility Decoupled Alignment (SUDA). SSR combines advantage-guided trajectory filtering and safe reference supervision to recover effective safety learning signals, while SUDA employs category-aware rewards and benign-only KL anchoring to mitigate over-refusal without compromising safety. Experiments across four Qwen3 and DeepSeek-R1-Distill variants and eight benchmarks show that our framework achieves the best average safety and reasoning performance, reducing average safety error by up to 32.83 percentage points while maintaining low over-refusal.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.23085 [cs.AI] |
| (or arXiv:2610.23085v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23085 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiawei Chen [view email]
[v1]
Fri, 16 Oct 2026 07:53:36 UTC (1,704 KB)
来源：arXiv cs.AI · arxiv.org