arXiv cs.CY· Guijia Zhang, Harry Yang·· 12 小时前AI 评分39
LLM-as-a-Judge 版本更替：换的是测量工具，不是标准
LLM-as-a-Judge Version Shift: The Instrument Changed, Not the Standard
AI 导读
研究用预注册冻结权重设计考察 Qwen、Gemma、Mistral 三个同族相邻版本更替是否保持 LLM 评审的测量标准，发现"新版本更严格"的假设不成立：两次更替偏向更宽松，第三次在锚点上方向相反。版本变化还会改变评分标准几何与分辨率，其中一次更替使 93% 的池内分数落在单一取值上。约 200 次调用的分层锚点审计可在部署前识别失效类型，全部 14882 次评审调用记录已公开。
正文
Abstract:Automated research pipelines increasingly rely on fixed LLM judges to rank, filter, and revise proposals, yet these judges are routinely replaced by newer versions. We study whether such replacements preserve the underlying measurement standard. Using a preregistered, frozen-weight design, we evaluate three adjacent same-family transitions across Qwen, Gemma, and Mistral on frozen ML research proposals under randomized information conditions. The common assumption that newer judges are simply harsher fails: two transitions shift toward greater leniency, while the third shifts in the opposite direction on anchors but attenuates off-anchor. More importantly, version changes can alter not only score location but also criterion geometry and resolution; one transition collapses clean-flawed discrimination, placing 93% of pool scores on a single value. Supplying identical facts does not close the version gap, placebo-controlled interventions show that observed packet effects are largely format-driven rather than content-driven, and global recalibration repairs the mean while explaining little object-level dispersion. We organize these failures with a measurement model in which a judge replacement can change its origin, criterion geometry, or resolution. A small stratified anchor audit of about 200 calls identifies the failure class before deployment, and frozen thresholds correctly diagnose two post-registration API judge replacements. We release six preregistration locks and the full audit trail of 14882 judge calls.
| Comments: | 25 pages, 5 figures, 14 tables |
| Subjects: | Computers and Society (cs.CY); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.32986 [cs.CY] |
| (or arXiv:2610.32986v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32986 arXiv-issued DOI via DataCite |
Submission history
From: Guijia Zhang [view email]
[v1]
Mon, 7 Sep 2026 08:30:21 UTC (184 KB)
来源：arXiv cs.CY · arxiv.org