arXiv cs.AI· Zihan Dong, Qishi Zhan, Minxuan Hu·· 10 小时前AI 评分38
研究：Agent 基准测试在任务耗尽前就已停止区分模型排名
Agent Benchmarks Stop Ranking Before They Run Out of Tasks
AI 导读
重评八个基准（含 SWE-bench Verified、Terminal-Bench）的已发布运行结果发现，多数 Agent 基准把一次运行压缩成单一 pass/fail 位，饱和更多来自测量方式而非任务本身。
正文 · 原文
Abstract:Data-science and ML-engineering agents are evaluated on benchmarks such as MLE-bench, ALE-Bench, SWE-fficiency and AlgoTune, whose tasks yield a quantity (a Kaggle metric, contest score or speedup). When one stops separating the best models, the usual remedy is a harder version. Re-scoring released runs of eight benchmarks, including SWE-bench Verified and Terminal-Bench for breadth, we find much of this saturation lies in the measurement, not the tasks. Most agent benchmarks reduce a run to one pass/fail bit, which fails in two ways. It is too coarse: neighbouring top-ten agents on SWE-bench Verified differ by less than its tests' errors, and correcting them reorders neighbours no more than deleting a random sixth of each agent's passes. It is capped at the reference: ALE-Bench's top ten, separated by contest score, tie when graded against the median human, although nearly half the runs beat that human. On fixed tasks, a higher bar only moves the problem; graded credit fixes it. We derive a clipped log-ratio score with dated anchors and cost as a separate axis. In a Terminal-Bench re-run, pass/fail-tied models differ over tenfold on the quantity the tests discard.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13541 [cs.AI] |
| (or arXiv:2611.13541v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13541 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zihan Dong [view email]
[v1]
Tue, 10 Nov 2026 04:03:04 UTC (153 KB)
来源：arXiv cs.AI · arxiv.org