arXiv cs.CL· Jhen-Ke Lin, Hong-Yun Lin·· 10 小时前AI 评分26
Balance of Benchmarks(BoB):兼顾证据保留与多重性的模型基准对比方法
Balance of Benchmarks: Evidence Retention and Multiplicity-Aware Model Comparison
AI 导读
研究者提出 Balance of Benchmarks(BoB),在保留更多基准观测的同时单独控制各基准的影响力:为每个基准拟合响应曲线以统一到共享能力尺度,并按描述相似度对密集邻域内的基准降权。
正文 · 原文
Abstract:Benchmark aggregation requires decisions about how much evidence to use and how much influence each benchmark receives. Curated suites control influence through selection, which can exclude informative observations. A broader pool can retain those observations. With equal weighting, repeated or closely related benchmarks gain influence as their number grows. We introduce Balance of Benchmarks (BoB) to retain eligible observations and control their influence separately. We fit benchmark-specific response curves to place different difficulty ranges on a shared ability scale and use description similarity to downweight benchmarks in crowded neighborhoods. We test retention and density weighting in separate interventions on WildScores, a sparse collection of 148 developer-reported benchmarks, using 14 Artificial Analysis benchmarks as a controlled reference. With the model population fixed and entire source-lineage families held out, suite expansion improves held-out prediction under BoB and equal-weight equating on both WildScores and BIG-bench. Under substantial reporting overlap, density weighting improves robustness to multiplicity. Complete re-listings yield smaller mean rank displacement under BoB (0.63 versus 1.95 for equal-weight equating), with benefits also in paraphrased re-listings and supported affine fits. The benefit can reverse if models report different benchmarks, because a single weight per column gives models that report only part of a group less than its intended share. We therefore compute each model's weights from the benchmarks it reports. This rule removes the reversal in controlled re-listings and keeps held-out prediction close to that of equal weighting. These comparisons separate evidence retention, benchmark influence, and reporting coverage and show why repeated entries need to be handled per model.
| Comments: | 40 pages. Revised analysis of evidence retention and per-model weighting. Includes appendix details and WildScores data |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.30044 [cs.AI] |
| (or arXiv:2608.30044v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.30044 arXiv-issued DOI via DataCite |
Submission history
From: Jhen-Ke Lin [view email]
[v1]
Sun, 30 Aug 2026 21:06:29 UTC (379 KB)
[v2]
Fri, 4 Sep 2026 19:47:26 UTC (291 KB)
[v3]
Thu, 17 Sep 2026 20:00:57 UTC (369 KB)
[v4]
Thu, 19 Nov 2026 16:51:40 UTC (3,677 KB)
来源:arXiv cs.CL · arxiv.org