arXiv cs.CL· Alicia Guerra, Jiaming Qu, Yibo Hu·· 11 小时前AI 评分37
评审者也是实验的一部分：审计 LLM 评审对同伴上下文的敏感性
The Judge is Part of the Experiment: Auditing Peer Context Sensitivity in LLM-Based Evaluation
AI 导读
一项控制实验发现，LLM 评审在评估多智能体系统时，同伴上下文不仅会改变智能体给出的答案，也会改变同一答案的评分，且不同评审的反应方向相反。研究覆盖 3 个 benchmark、4 个开源权重模型作为智能体和 6 个评审，错误答案的同伴上下文对智能体的误导大于正确答案的帮助，评审在缺少正确答案时难以区分已知正确与已知错误的答案。
正文 · 原文
Abstract:Large language model (LLM)-based judges are increasingly used to evaluate AI systems. An underexplored area is applying these judges to multi-agent systems, where they may see the interactions behind a final answer. These interactions, which we call peer context, can affect an evaluation in two ways: (1) they can change the answer produced by the multi-agent system, because an agent might be swayed by other agents, and (2) they can change how a judge evaluates that same answer, depending on whether the judge sees them. In this paper, we separate these two effects through a controlled experiment. We first let an agent answer a question and then answer again after seeing peer responses. We then prompt judges to rate the same final answer with and without the peer context. Finally, we check whether the judges' ratings establish a valid measure of answer quality. Across three benchmarks, four open-weight LLMs as agents, and six judges, we find that peer context changes not only the answer but also how an identical answer is judged, and judges react to it in opposite directions. Moreover, providing peer context for a wrong answer misleads agents more than correct peers help them; without the correct answer, judges struggle to tell known-correct from known-wrong answers---a problem that a well-fitted statistical model does not reveal. Our results show that even when evaluating the same answer, how a judge is set up can change the conclusions differently. We therefore call for evaluations to report what each judge sees, document detailed results per judge, and validate rating scales. These practices are an important step toward the accountable use of LLM-based judges.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.04463 [cs.CL] |
| (or arXiv:2608.04463v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04463 arXiv-issued DOI via DataCite |
Submission history
From: Alicia Guerra [view email]
[v1]
Wed, 5 Aug 2026 05:38:34 UTC (205 KB)
[v2]
Thu, 13 Aug 2026 17:18:31 UTC (205 KB)
[v3]
Wed, 4 Nov 2026 20:41:21 UTC (308 KB)
来源：arXiv cs.CL · arxiv.org