arXiv cs.AI· Sida I. Wang·· 3 天前AI 评分23
如何测量 LLM 评测中的所有噪声
Measuring all the noises of LLM Evals
AI 导读
研究定义并测量了 LLM 评测中的三类噪声：同一问题生成不同答案的预测噪声、采样问题的数据噪声，以及依据总方差定律合并的总噪声。作者提出 all-pairs comparison 方法，对 LLM 所有配对做成对比较，基于数百万条问题级预测测量各噪声分量。结果显示多数评测在所有模型对上呈现高度可预测的总噪声水平，且成对预测噪声通常大于成对数据噪声，通过平均降低预测噪声可显著提升统计功效。
正文 · 原文
Abstract:Applying well-established statistical methods effectively to LLM evals requires consideration of model correlations and prediction noises. Leading evals often ignore data noise, have low statistical power, or make unnecessary trade-offs. We define and measure three types of noise: 1) prediction noise from generating different answers on a given question, 2) data noise from sampling questions, and 3) their combined total noise following the law of total variance. To emphasize relative comparisons and gain statistical power, we propose \emph{all-pairs comparison}, which applies paired comparison to all pairs of LLMs and measures all the noise components based on millions of question-level predictions across many evals and settings, revealing clear trends. First, most evals exhibit a characteristic and highly predictable total noise level across all model pairs. Second, paired prediction noise typically exceeds paired data noise, which means reducing prediction noise by averaging can significantly increase statistical power. By measuring all the noises together, we can assess eval results in context and make better empirical decisions.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (stat.ML) |
| Cite as: | arXiv:2512.21326 [cs.LG] |
| (or arXiv:2512.21326v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.21326 arXiv-issued DOI via DataCite |
Submission history
From: Sida I. Wang [view email]
[v1]
Wed, 24 Dec 2025 18:54:37 UTC (2,926 KB)
[v2]
Sun, 29 Mar 2026 00:57:41 UTC (2,938 KB)
[v3]
Tue, 13 Oct 2026 04:11:28 UTC (3,942 KB)
来源：arXiv cs.AI · arxiv.org