arXiv cs.CL· Sikun Xu, Raphael Thomadsen, Dennis J. Zhang, Heng Zhang·· 9 小时前AI 评分36
SOTA 还是运气?LLM 排行榜中的“赢家诅咒”
SOTA or Luck? The Winner's Curse in LLM Leaderboards
AI 导读
研究指出公开 LLM 排行榜上排名靠前的模型性能可能被系统性高估,这种“赢家诅咒”源于排名噪声基准分数时把采样运气也算作真实能力,并由此产生贯穿榜单的排名诱导偏差。
正文 · 原文
Abstract:Public LLM leaderboards guide model evaluation and adoption, yet the reported performance of top-ranked models can be systematically overstated. This leaderboard winner's curse arises because ranking noisy benchmark scores rewards sampling luck alongside genuine ability. The same selection effect can distort scores throughout the leaderboard, a phenomenon we call the rank-induced bias. We develop an $m$-out-of-$N$ bootstrap to correct for this bias. It assumes a fixed model pool with independent, identically distributed bounded tasks while allowing arbitrary within-task correlation among models. Under exact population tiers separated by positive gaps, we prove that the bootstrap reduces $\Theta(N^{-1/2})$ ranking bias to expected residual bias $o(N^{-1/2})$. Across 14 benchmarks from 5 benchmark families, the rank-1 adjustments range from approximately 0.09 to 3.86 percentage points. In our SWE-bench Verified snapshot, the tied leaders receive an adjustment of 0.90 percentage points. Such overstatement is large enough to make the apparent winner statistically fragile. Thus, we use the same bootstrap to further quantify the intervals of plausible rank positions for each model and group models into tiers. We establish sufficient conditions for consistent recovery of population tiers. On SWE-bench Verified, this procedure places 13 of 173 models in the leading tier. We combine the score adjustment and rank intervals in the Rank Uncertainty and Leaderboard Evaluation Report (RULER). We validate its finite-sample performance using sample splitting and calibrated simulations, and advocate publishing RULER alongside raw rankings so readers can assess both reported performance and the precision of the ordering. RULER also provides guidance for benchmark design that accounts for selection effects and ranking uncertainty.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Methodology (stat.ME); Machine Learning (stat.ML) |
| Cite as: | arXiv:2611.22227 [cs.LG] |
| (or arXiv:2611.22227v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22227 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sikun Xu [view email]
[v1]
Tue, 17 Nov 2026 05:18:14 UTC (130 KB)
来源:arXiv cs.CL · arxiv.org