研究：LLM-as-a-Judge 评测可靠性不等于有效性，21 个评审模型在一致性、偏差审计中暴露系统性缺陷

This paper has been withdrawn by arXiv Admin

No PDF available, click to view other formats

Abstract:LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.

Comments:	arXiv admin note: This submission has been withdrawn because it does not meet arXiv's research content quality standards
Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2606.19544 [cs.CL]
	(or arXiv:2606.19544v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2606.19544 arXiv-issued DOI via DataCite

Submission history

From: arXiv Admin [view email]
[v1] Wed, 17 Jun 2026 19:37:13 UTC (506 KB)
[v2] Tue, 3 Nov 2026 20:53:09 UTC (1 KB) (withdrawn)