arXiv cs.AI· Jeremy Canale, Fabien Soulis·· 1 天前AI 评分44
ReplayRCA：用反事实重放评测智能体失败的事后诊断与实时监督
ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay
AI 导读
ReplayRCA 基准以“因果修复”分数评测智能体失败诊断：将环境恢复到出错步骤前，用诊断模型提出的动作替换原动作后让同一智能体继续，得分是续跑成功率减去配对安慰剂组的差值。
正文 · 原文
Abstract:When a language-model agent fails a multi-step task, can another model find the decisive mistake and fix it with a single action? We introduce ReplayRCA, a benchmark that scores such diagnoses by intervention rather than by agreement with annotations: the environment is restored to the state preceding the chosen step, the proposed action replaces the agent's own, and the same agent continues. The score, causal repair, is the continuation success rate minus that of a paired placebo that replays the agent's original action through the same mechanism. Two tracks share this protocol: a post-hoc autopsy, where the diagnoser sees the whole failed run and is told that it failed, and live oversight, where a supervisor watches the run unfold and may intervene once, before an irreversible action, without knowing whether the run will fail. Tasks come from OpsWorld, a deterministic, procedurally generated back-office environment with six difficulty levels, the last of which is multimodal (QR codes, handwriting). On 62 calibrated systematic failures, the best of 13 frontier models (Grok 4.7) reaches a causal repair of +51.3 points; the top eight models are statistically indistinguishable, the flagship models of the two largest vendors rank 8th and 12th, and 5 items are repaired by no model. Forcing a strong model to intervene at the last step costs it 17.1 points, so the last-step shortcut of annotation-based attribution benchmarks does not transfer. On 87 runs, the two best live supervisors are statistically tied and one of them is a small open-weight model (DeepSeek V4.1 Flash); the supervisor that is among the best post-hoc diagnosers (GPT-6.1 Sol) is the weakest live one (-14.5 points on the same runs, p = 0.002), so post-hoc diagnostic performance can be a poor proxy for live oversight. The environment and the evaluation code are released at this https URL.
| Comments: | 29 pages, 10 figures, 16 tables. Code: this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41025 [cs.AI] |
| (or arXiv:2610.41025v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41025 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fabien Soulis [view email]
[v1]
Thu, 29 Oct 2026 22:11:00 UTC (415 KB)
来源：arXiv cs.AI · arxiv.org