MonitorBench：面向大语言模型思维链可监控性的综合基准
MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models
研究者提出 MonitorBench，一个系统评估 LLM 思维链可监控性的开源基准，包含覆盖 7 大类 19 项任务的 1,540 个测试实例，并设计两种提示词压力测试设置来量化可监控性的下降程度。
Authors:Han Wang, Yifan Sun, Brian Ko, Mann Talati, Jiawen Gong, Zimeng Li, Naicheng Yu, Xucheng Yu, Wei Shen, Vedant Jolly, Huan Zhang
Abstract:Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the actual reasons (i.e., decision-critical factors) driving the model's behavior, leading to the reduced CoT monitorability problem. This limits the use of CoTs for reliable oversight. However, a comprehensive and fully open-source benchmark for thoroughly evaluating CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,540 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two prompting stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments show that CoT monitorability is a conditional property affected by the evaluated LLM, monitor LLM, and task characteristics. Across these factors, monitorability is higher when decision-critical factors shape the intermediate reasoning process, rather than merely influencing the final answer. Under stress-test prompting, most evaluated LLMs can intentionally reduce monitorability, mainly on tasks where decision-critical factors are not structurally required by the reasoning process. Overall, MonitorBench provides a basis for further research on AI control, reasoning faithfulness, stress-test monitorability, and monitoring scaffords. The code is available at this https URL.
| Comments: | COLM 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.28590 [cs.AI] |
| (or arXiv:2603.28590v5 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2603.28590 arXiv-issued DOI via DataCite |
Submission history
From: Han Wang [view email]
[v1]
Mon, 30 Mar 2026 15:37:42 UTC (10,282 KB)
[v2]
Thu, 2 Apr 2026 02:22:56 UTC (14,389 KB)
[v3]
Sun, 9 Aug 2026 16:17:33 UTC (6,924 KB)
[v4]
Mon, 28 Sep 2026 14:42:45 UTC (6,922 KB)
[v5]
Wed, 11 Nov 2026 01:28:56 UTC (6,918 KB)
来源：arXiv cs.AI · arxiv.org