arXiv cs.CL· Bruno Brocai, Maria Becker·· 12 小时前AI 评分31
配对难度很重要：重新审视成对 LLM-as-a-Judge 评估与一致性
Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
AI 导读
研究指出，位置偏差、传递性和成对一致性这三项 LLM-as-a-Judge 常用可靠性指标具有误导性：在 Bradley–Terry 几何下，它们主要由排名差距小的配对主导，而这类配对的不一致在信息论上本就可预期。受控模拟和两个人工评分语料验证显示，这些指标与排名准确率仅弱相关，其预测成分集中在差距大的配对区间。作者主张按排名差距条件化评估评判模型，并最好以人类排名为基准。
正文 · 原文
Abstract:Large Language Model judges are widely used to rank texts and text-generating systems through pairwise comparison, and their reliability is typically assessed via three proxies: position bias, transitivity, and pairwise agreement (self- or human-labeled). Because these proxies drive judge selection and benchmarking, a substantial literature reporting that judges perform poorly on them risks steering practitioners away from otherwise capable evaluators. We argue this assessment is misleading. Under the Bradley--Terry geometry underlying pairwise aggregation, each proxy is dominated by close-rank-gap pairs, where inconsistency is information-theoretically expected and individual verdicts contribute little to the aggregate ranking; far-gap pairs carry the ranking signal but barely move the proxies. We formalize this argument and validate it in a controlled simulation and on two human-rated corpora: the proxies correlate only weakly with ranking accuracy against gold, and their predictive component concentrates in the far-gap regime. Judges should therefore be assessed on rank-gap-conditional metrics, ideally against human rankings. Code at this https URL.
| Comments: | Accepted as an EMNLP 2026 short paper. v2: improve graph labels |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.37577 [cs.CL] |
| (or arXiv:2609.37577v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.37577 arXiv-issued DOI via DataCite |
Submission history
From: Bruno Brocai [view email]
[v1]
Tue, 29 Sep 2026 13:40:25 UTC (365 KB)
[v2]
Tue, 20 Oct 2026 14:15:16 UTC (369 KB)
来源：arXiv cs.CL · arxiv.org