arXiv cs.CL· Ziyi Zhu, Olivier Tieleman, Alexey Bukhtiyarov, Jinghong Chen·· 12 小时前AI 评分32
CyclicJudge：在 LLM 评测中高效缓解评审模型偏差
CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
AI 导读
CyclicJudge 通过将评审模型轮转分配给各场景，在固定评审面板和评审调用预算下抵消评审模型的平均加性偏移，同时保持与单评审评测相同的成本。该方法基于将 benchmark 分数方差分解为场景、生成、评审和残差四部分的分析，在 MT-Bench 和 MindEval 上的实验验证了其降低分数方差、稳定模型排名的效果。
正文 · 原文
Abstract:LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be averaged out by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. We introduce a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges to scenarios, is demonstrated to be the optimal strategy for a fixed judge panel and judge-call budget: it cancels the average additive judge offsets while matching the cost of single-judge evaluation. Empirical results on MT-Bench and MindEval validate the effectiveness of CyclicJudge as predicted, across both general-purpose and domain-specific evaluation settings, reducing benchmark score variance and stabilizing the resulting model ranking.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.01865 [cs.CL] |
| (or arXiv:2603.01865v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.01865 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 4527-4539, Budapest, Hungary, October 2026 |
| Related DOI: | https://doi.org/10.18653/2026.emnlp-main.197
DOI(s) linking to related resources |
Submission history
From: Ziyi Zhu [view email]
[v1]
Mon, 2 Mar 2026 13:46:32 UTC (59 KB)
[v2]
Fri, 6 Mar 2026 19:05:47 UTC (72 KB)
[v3]
Mon, 4 May 2026 15:49:28 UTC (73 KB)
[v4]
Tue, 3 Nov 2026 21:24:14 UTC (80 KB)
来源：arXiv cs.CL · arxiv.org