arXiv cs.CL· Changyi Li, Jiayi Wang, Xudong Pan, Geng Hong, Min Yang·· 6 天前AI 评分42
ReasoningShield：面向大型推理模型推理轨迹的安全检测基准与轻量模型
ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
AI 导读
针对大型推理模型（LRM）的思维链（CoT）推理轨迹中可能隐藏有害内容的问题，研究者提出 ReasoningShield 基准及配套轻量模型。
正文 · 原文
Abstract:Large Reasoning Models (LRMs) leverage explicit reasoning traces, known as Chain-of-Thought (CoT) reasoning, to decompose complex problems into intermediate steps before deriving final answers. However, these reasoning traces introduce unique safety challenges: harmful content can be embedded in intermediate steps even when final answers appear benign. Our study reveals that existing state-of-the-art moderation tools experience significant performance degradation on CoT moderation, with F1 scores dropping by up to 35.2% compared to traditional answer moderation. To address these challenges, we present ReasoningShield, a comprehensive benchmark and a suite of strong lightweight models for CoT safety moderation. Our benchmark includes 9.2K annotated (Query, CoT) pairs with structured stepwise risk analysis, consisting of a 7K-sample training set and a 2.2K human-annotated test set, spanning 10 risk categories, 3 safety levels, and 8 LRMs with diverse reasoning paradigms. To further demonstrate the utility of the benchmark, we develop lightweight models with 1B/3B parameters using a two-stage training strategy. Our models achieve 91.8% F1 on our benchmark, substantially outperforming leading tools such as LlamaGuard-4 by 35.6% and commercial models such as GPT-4o by 15.8%, while generalizing effectively across diverse reasoning paradigms and unseen scenarios. Resources are publicly available: Project Page: this https URL, Code: this https URL, Models: this https URL, and Dataset: this https URL.
| Comments: | Accepted at NeurIPS 2026, Evaluations & Datasets Track. Resources are publicly available: Project Page: this https URL, Code: this https URL, Models: this https URL, and Dataset: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2505.17244 [cs.CL] |
| (or arXiv:2505.17244v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2505.17244 arXiv-issued DOI via DataCite |
Submission history
From: Changyi Li [view email]
[v1]
Thu, 22 May 2025 19:44:41 UTC (3,351 KB)
[v2]
Wed, 15 Oct 2025 11:47:26 UTC (2,260 KB)
[v3]
Fri, 9 Oct 2026 08:35:15 UTC (4,813 KB)
来源：arXiv cs.CL · arxiv.org