跳到正文
arXiv cs.AI· Yezhou Cheng, Runjia Du, Zeming Liu, Qibai Chen, Hang Lyu, Yilan Wei, Yankai Zeng, Bojun Lin·· 10 小时前AI 评分31

固定预算下窄置信区间的 LLM 评测框架

Cost-Efficient LLM Evaluation: A Framework for Narrow Confidence Intervals Under Fixed Budgets

AI 导读

研究者提出一个在固定评测预算下分配任务与重复运行的框架,利用同一任务多次运行间的不一致来约束估计误差,从而在相同置信水平下获得更窄的置信区间。该框架在任务与重复预算增长时达到理论最优的收窄速率,并给出全覆盖与可跳过任务两种情形下的匹配下界。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large language models (LLMs) play an increasingly important role in both academic research and industry. Reliable evaluation helps researchers understand model capabilities and assess progress during model development. However, evaluation can be costly, and researchers often need to assess performance within a fixed budget. Because model outputs can vary across runs, this budget must be allocated between testing different tasks and repeating selected tasks. A confidence interval quantifies the uncertainty in the resulting performance estimate. Obtaining narrower intervals at the same confidence level and evaluation cost provides a more precise assessment of model performance, helping researchers draw clearer conclusions from limited observations. We introduce a framework that allocates a fixed evaluation budget across tasks and repeated runs, with or without full task coverage. The framework then uses disagreements between runs on the same task to bound estimation error, allowing narrower confidence intervals at the same confidence level when the sampled repeats show little variability. When repeated runs on each task yield the same outcome, we prove that our framework achieves the fastest theoretically possible rates for narrowing confidence intervals in the worst case as the number of tasks and the budget for repeats grow. We establish matching lower bounds both when every task must be covered and when tasks may be skipped, even for evaluators who use earlier results to decide which run to carry out next. On LiveCodeBench, our framework yielded narrower confidence intervals on average than uniform random sampling in 15 of 16 model configurations, with a median width reduction of 30.6% at the same confidence level and the same budget of 1,100 outputs per configuration.
Subjects: Artificial Intelligence (cs.AI); Information Theory (cs.IT)
Cite as: arXiv:2609.29140 [cs.AI]
  (or arXiv:2609.29140v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2609.29140

arXiv-issued DOI via DataCite

Submission history

From: Yezhou Cheng [view email]
[v1] Thu, 24 Sep 2026 07:15:07 UTC (157 KB)
[v2] Wed, 18 Nov 2026 01:01:30 UTC (55 KB)

来源:arXiv cs.AI · arxiv.org