arXiv cs.CL· Jungsoo Park, Ethan Mendes, Gabriel Stanovsky, Alan Ritter·· 3 天前AI 评分24
语言模型的预期评估：从任务描述预测 LLM Benchmark 分数
Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions
AI 导读
研究者提出"预期评估"（anticipatory evaluation）设定，并发布 EVALCAST——一个包含 1,519 篇 arXiv 论文中 2,290 组描述-性能配对的数据资源，测试集包含模型知识截止后发表的论文。实验显示，前沿 LLM 通过回归预测绝对性能仍具挑战，而配对预测在匹配设定下可恢复相对性能排序；在截止后与流式评估上的表现相当，表明该信号并非仅来自已报道结果。
正文 · 原文
Abstract:Modern ML research increasingly faces an evaluation bottleneck: building benchmarks and running experiments is costly, slowing iteration. We ask how much can be inferred about evaluation outcomes before experiments, given only a natural-language description of the task and setup. We call this setting anticipatory evaluation. We introduce EVALCAST, a literature-grounded resource of 2,290 description-performance pairs from 1,519 arXiv papers, including a test set of papers published after the model's knowledge cutoff. We evaluate two tasks: predicting absolute performance through regression and comparing benchmark difficulty through pairwise prediction. Experiments with frontier LLMs show that accurate regression remains challenging, while pairwise predictions can recover relative performance ordering under matched settings. Comparable performance on post-cutoff and streaming evaluations suggests that the observed signal is not solely attributable to previously reported results. Overall, EVALCAST provides a resource and evaluation framework for studying anticipatory evaluation and its potential use in prioritizing benchmarks.
| Comments: | 29 pages, 6 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.20645 [cs.CL] |
| (or arXiv:2509.20645v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2509.20645 arXiv-issued DOI via DataCite |
Submission history
From: Jungsoo Park [view email]
[v1]
Thu, 25 Sep 2025 01:02:27 UTC (906 KB)
[v2]
Mon, 1 Dec 2025 20:11:55 UTC (1,050 KB)
[v3]
Wed, 4 Feb 2026 16:21:49 UTC (1,491 KB)
[v4]
Tue, 13 Oct 2026 18:26:56 UTC (962 KB)
来源：arXiv cs.CL · arxiv.org