arXiv cs.AI· Guoyang Xu, Rui Zhang, Xiuyu Wu, Hao Chen·· 2 天前AI 评分44
编码智能体的报告遗漏了什么：CTXGAP 测量执行上下文与报告之间的任务状态差距
What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports
AI 导读
CTXGAP 测量研究发现，编码智能体在按需回答任务状态问题时准确率达 93.6%，但仅读报告的监督者只能恢复 52.0%，与全上下文读者的 92.7% 相差 40.7 个百分点。在 30 个已验证修复的最终检查点上差距仍有 34.8 个百分点，完整终端回滚可将恢复率提升至 73.6%，但仍不及全上下文。该研究将基于报告的任务状态可恢复性确立为一项补充评估维度。
正文 · 原文
Abstract:Human supervisors rely on coding agents' reports to assess progress, verify repairs, and decide when to intervene. But a supervisor cannot investigate a failure they were never told occurred. Failed attempts, workarounds, and unresolved risks may disappear from routine reports, leaving supervisors unaware of issues that warrant scrutiny. We conduct CTXGAP, a measurement study quantifying the gap in task-state recoverability between agents' accumulated user-facing reports and their full recorded execution contexts. Across 37 coding tasks and 4,799 valid probes, agents answer 93.6% of task-state questions on demand; full-context readers recover 92.7%, while report-only readers recover 52.0%, a 40.7-point gap. At the final checkpoints of 30 verified fixes, the gap remains 34.8 points. Human annotation identifies states mentioned without enough supporting detail as the largest category of unrecovered information, and model-assessed supervision-relevant states retain substantial gaps. Full terminal scrollback improves recovery to 73.6% but still trails full context. Under matched report lengths, context-informed rewrites improve recovery by about eight points, without a detectable targeted-over-natural advantage. Positive gaps persist across harnesses and independent DeepSeek execution and measurement. CTXGAP establishes report-based task-state recoverability as a complementary evaluation dimension.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20003 [cs.AI] |
| (or arXiv:2610.20003v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20003 arXiv-issued DOI via DataCite |
Submission history
From: Rui Zhang [view email]
[v1]
Wed, 14 Oct 2026 11:20:15 UTC (2,963 KB)
[v2]
Thu, 15 Oct 2026 08:51:29 UTC (2,965 KB)
来源：arXiv cs.AI · arxiv.org