arXiv cs.AI· Seanie Lee, Sangwoo Park, Yumin Choi, Gyeongman Kim, Minki Kang, Jihun Yun, Sung Ju Hwang, Dongmin Park, Jongho Park·· 11 小时前AI 评分42
ThinkSafe：推理模型的自生成安全对齐方法
ThinkSafe: Self-Generated Safety Alignment for Reasoning Models
AI 导读
ThinkSafe 无需外部教师模型，直接从冻结的学生模型生成安全对齐数据，通过拒绝引导激发安全推理并用安全护栏过滤响应后离线微调。在 DeepSeek-R1-Distill 和 Qwen3 上实验显示安全性能大幅提升且推理表现保持竞争力；Qwen3-0.6B 上以约七分之一的实际耗时达到优于 GRPO 的安全效果和相当推理水平。代码、模型和数据集已开源。
正文 · 原文
Abstract:Large reasoning models achieve strong performance through reinforcement learning but can suffer safety degradation. External teacher distillation can restore safety yet may compromise reasoning performance. We formalize safety realignment as a KL projection: the student's native filter-accepted distribution is uniquely KL-optimal, while teacher targets incur excess KL when they differ. Guided by this analysis, we propose ThinkSafe, which generates safety-alignment data from the frozen student without external teachers. Refusal steering elicits safety reasoning, and a safety guard filters the responses before offline fine-tuning. Our analysis bounds steering-induced target distortion and gives a sufficient condition for lower average target KL than teacher distillation. Experiments on DeepSeek-R1-Distill and Qwen3 show substantial safety improvements with competitive reasoning performance. On Qwen3-0.6B, ThinkSafe achieves better safety and comparable reasoning to GRPO in about one-seventh of the wall-clock time, including data generation. Code, models, and datasets are available at this https URL and this https URL.
| Comments: | NeurIPS 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.23143 [cs.AI] |
| (or arXiv:2601.23143v5 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2601.23143 arXiv-issued DOI via DataCite |
Submission history
From: Seanie Lee [view email]
[v1]
Fri, 30 Jan 2026 16:31:02 UTC (605 KB)
[v2]
Fri, 8 May 2026 17:26:01 UTC (606 KB)
[v3]
Tue, 12 May 2026 15:49:44 UTC (597 KB)
[v4]
Wed, 13 May 2026 02:16:49 UTC (597 KB)
[v5]
Fri, 30 Oct 2026 10:17:31 UTC (1,218 KB)
来源：arXiv cs.AI · arxiv.org