arXiv cs.AI· Tianyang Xu, Jing Gao·· 3 天前AI 评分26
审计失败归因：标签可预测性与评判模型行为
Auditing Failure Attribution: Label Predictability and Judge Behavior
AI 导读
针对失败归因基准中 LLM 评判模型可能依赖表面线索而非跨步骤核查的问题，研究者提出输入限制、线索跟随、残差准确率和聚合四项测试。在三个基准上，简单预测器即可匹敌或超越 GPT-4.1；在 Who&When Pro 系统中，仅读取单步推理的分类器能找出 95-100% 的故障步骤，而注入错误后微调评判模型虽领先 GPT-4.1 达 41 分，但加入任务推理后骤降 31-99 分。
正文 · 原文
Abstract:Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure. A high score should mean the judge checked the faulty step against the run (a cross-step check). However, spurious signals (surface cues) that predict the annotation can produce a high score. In this paper, we propose four tests to separate checking from cue use. Input restriction asks whether partial input predicts the annotation. Cue following checks whether moved cues change answers. Residual accuracy scores judges where cue-only predictors fail. Aggregation asks whether the failure mix or metric changes rankings. Real observations motivate these tests: on three benchmarks, simple predictors match or beat GPT-4.1. In two Who&When Pro systems, a classifier reading one step's reasoning finds 95-100% of faulty steps, motivating input restriction. In one of them, error injection leaves no reasoning after the faulty step; judges fine-tuned on its labels beat GPT-4.1 by 41 points. Adding task-related reasoning after that step moves this cue: fine-tuned judges lose 31-99 points, but zero-shot judges lose at most 9, motivating cue following. On Aegis, blaming every agent matches GPT-4.1 on micro-F1 but trails by 19 points in exact-set accuracy, motivating aggregation. We recommend reporting these tests when developing failure-attribution benchmarks.
| Comments: | 26 pages |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.17747 [cs.CL] |
| (or arXiv:2610.17747v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17747 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianyang Xu [view email]
[v1]
Tue, 13 Oct 2026 05:26:33 UTC (142 KB)
来源：arXiv cs.AI · arxiv.org