arXiv cs.AI· Jiale Liu, Huajun Xi, Shaokun Zhang, Yifan Zeng, Tianwei Yue, Chi Wang, Jian Kang, Qingyun Wu, Huazheng Wang·· 6 天前AI 评分22
Who&When Pro：LLM 真能定位 AI 智能体的失败吗？
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
AI 导读
研究者推出 Who&When Pro，一个面向 AI 智能体自动失败归因的大规模 benchmark，通过严格受控流程在完整回放成功前缀后才注入失败，构建了覆盖 3 种模态、26 个 benchmark 的 12,326 条带金标签失败轨迹。论文还开展大量实验，揭示模型在不同模态、协议和模型族间归因失败的系统性模式，为后续自动失败归因系统提供实证指导。
正文 · 原文
Abstract:Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.
| Subjects: | Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2607.09996 [cs.AI] |
| (or arXiv:2607.09996v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09996 arXiv-issued DOI via DataCite |
Submission history
From: Jiale Liu [view email]
[v1]
Fri, 10 Jul 2026 21:45:16 UTC (2,480 KB)
[v2]
Fri, 9 Oct 2026 04:04:24 UTC (2,462 KB)
来源：arXiv cs.AI · arxiv.org