跳到正文
arXiv cs.AI· Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai·· 23 小时前AI 评分42

编码智能体何时该停?SWE-bench Verified 上 6000 次运行的 anytime 评估

Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop

AI 导读

研究通过 anytime 评估回放约 6000 次运行,在 SWE-bench Verified 上测试 8 个开源模型、2 种 scaffold,发现自测通过后继续工作的价值差异巨大,可节省成本从不足 1% 到 57%。"同一补丁两次通过即停止"的规则在预注册复现中节省 23%,但预注册质量测试未确立非劣性;微调智能体模仿该规则在 6 个随机种子上节省 30%,质量保持仍不确定。

正文 · 原文

Authors:Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai

View PDF HTML (experimental)

Abstract:Coding agents often continue working after their own tests pass. Does this additional work improve the patch, and how much could be saved by stopping earlier? We study these questions through anytime evaluation: replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step. For policies that only stop a run or select an earlier patch, this allows paired comparisons without generating new trajectories, provided replay preserves the relevant states. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, we find substantial differences in the value of continued work. The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority. We explain these differences by measuring how often a rule fires, how often the patch changes afterwards, and whether those changes repair or break it. Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain. Our results help identify both the opportunities and the risks of early stopping.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22901 [cs.AI]
  (or arXiv:2611.22901v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22901

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zerui Yang [view email]
[v1] Tue, 17 Nov 2026 15:07:35 UTC (521 KB)

来源:arXiv cs.AI · arxiv.org