arXiv cs.AI· Bingyu Shen, Jiaqi Tang, Lan Wei, Yi Sheng, J. Jenny Li, Boyang Li·· 3 天前AI 评分30
可执行前缀价值：定位 LLM 智能体失败步骤的新方法
Executable Prefix Values for Locating Agent Failures
AI 导读
研究提出用可执行前缀价值 V(h_t) 定位 LLM 智能体失败点，在 150 个 SWE-bench Verified 任务上为 47 和 60 条失败轨迹构建成功概率曲线并标定不可逆点（PNR）。
正文 · 原文
Abstract:Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. We ask whether such labels point at causes or symptoms, and make the distinction executable: we restart the agent from reconstructed intermediate states of a failed trajectory and estimate its prefix value V(h_t), the probability of success from each step, with a certified point of no return (PNR) for a failure. On 150 SWE-bench Verified tasks with two frozen open-weight agents, we build such curves for 47 and 60 failed trajectories and probe every step labeled by four frontier judges under the Traverse rubric. We find that the labels do not mark what a controller needs. Judges of different lineages agree on the exact step in only 21/57 cases, and the newest judges flag mistakes in 8/10 and 10/10 successful runs. For 66% of the labels, the agent's success probability does not drop at the labeled step; only 7/188 answers name the decisive step. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline, and real restarts confirm the advantage, whereas restarting at a judge's label does not (-0.08 to -0.03). We release our code, curves, and labels.
| Comments: | 46 pages, 12 figures, 60 tables |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.17633 [cs.AI] |
| (or arXiv:2610.17633v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17633 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bingyu Shen [view email]
[v1]
Tue, 13 Oct 2026 03:00:35 UTC (661 KB)
来源：arXiv cs.AI · arxiv.org