arXiv cs.AI· Drake Benjamin Brown, Khushbu Pahwa, Yubin Ge, Salvatore Romeo, Akhil Agnihotri, Yi Zhang, Xiaofei Ma, Daniele Bonadiman·· 11 小时前AI 评分49
研究揭示 Agent-as-a-Judge 的确认偏误：一条虚假预筛备注可推翻 40% 正确判定
Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge
AI 导读
一项 arXiv 预印本研究显示，Agent-as-a-Judge 即使自行收集到决定性证据，仍会因一条被归因于权威、与正确判定无关的虚假预筛备注而推翻最多 40% 的正确判定，覆盖四个 judge 与五个数据集。研究将行为变化归为过度信任、幻觉和调查不足三种失败模式，两个 Claude judge 的有害翻转中 67-75% 属过度信任，轻量缓解措施只能削弱而无法消除该偏误。
正文 · 原文
Abstract:Evaluating open-ended agent tasks increasingly falls to an Agent-as-a-Judge: an evaluator handed tools and the agent's full trajectory so it can verify a result against the environment. The premise is that a judge that gathers its own evidence should be grounded by it. We show that this grounding does not immunize the judge against confirmation bias: a single false pre-screening note, attributed to an authority and carrying no evidential bearing on the correct verdict, overturns up to 40% of a tool-using judge's correct verdicts across four judges and five datasets, even though the judge collects decisive evidence. The shift is driven by specific properties of the cue. We treat this cue as a controlled diagnostic of confirmation bias rather than an adversarial attack. Our central contribution is to explain how confirmation bias changes the judge's behavior in a trajectory. We classify changes in judge behavior and attribute changes to one of three failure modes: over-trust (verify, conclude correctly, then defer), hallucination (fabricate support for the flip), and under-investigation (stop at the first corroborating signal). For the two Claude judges, the failure is predominantly over-trust (67-75% of harmful flips), while smaller judges have error modes spread across all three. Lightweight mitigations blunt but never remove the bias.
| Comments: | Preprint. Under Review |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05662 [cs.AI] |
| (or arXiv:2611.05662v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05662 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Khushbu Pahwa [view email]
[v1]
Tue, 3 Nov 2026 21:47:19 UTC (958 KB)
来源：arXiv cs.AI · arxiv.org