arXiv cs.CL· Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang·· 13 小时前AI 评分38
基准测试是对难度的经验测度：NeurIPS 2026 论文提出诊断指标与 BenchFill 方法
Your Benchmark Is an Empirical Measure Over Difficulty
AI 导读
一篇被 NeurIPS 2026 接收的论文提出将基准测试视为对难度的经验测度，并形式化了两类失效模式：测度错配（基准难度分布与目标难度分布不一致）和覆盖缺口（难度轴上未覆盖区域使基准无法检测局部训练进展）。
正文 · 原文
Abstract:A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation. This perspective allows us to identify and formalize two central failure modes: measure mismatch, where the benchmark difficulty distribution differs from the intended difficulty distribution, and coverage gaps, where uncovered regions along the difficulty axis prevent benchmarks from detecting meaningful localized training progress. We propose two diagnostic metrics that capture these two failure modes and provide rigorous theoretical guarantees. Empirically, we analyze 25 popular benchmarks spanning 7 domains using carefully estimated item difficulties. Our results show that many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis uncovered, limiting their ability to monitor the model improvement process. Finally, we turn diagnosis into action by introducing BenchFill, a simple difficulty-targeted item rewriting algorithm that effectively reduces local coverage gaps.
| Comments: | Accepted at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.29927 [cs.LG] |
| (or arXiv:2610.29927v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29927 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yifan Sun [view email]
[v1]
Wed, 21 Oct 2026 16:02:37 UTC (966 KB)
来源：arXiv cs.CL · arxiv.org