arXiv cs.AI· Yuanzhe Liu, Bastian Lipka, Avraham Shinnar, Jatin Ganhotra, Yao Ma, Martin Hirzel·· 12 小时前AI 评分31
编码基准测试如何避免提示泄露与猜测：SWE-bench Pro 等基准的过度指定问题研究
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
AI 导读
研究发现 SWE-bench Pro、FEA-bench、Doc2Feat-bench 和 Multi-SWE-bench 通过从标准答案中提取提示来补充任务描述，反而造成任务过度指定，让解法得以绕过基准本应考察的困难步骤。作者通过不同指定程度的实验，在“剧透”与“猜测”之间寻找平衡，并提出推导精简提示的技术方案，相关提示将公开发布。
正文 · 原文
Abstract:Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking. Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure. Since future-looking benchmarks should be difficult, over-specification reduces a benchmark's shelflife of community adoption. Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification. For instance, SWE-bench Pro, FEA-bench, Doc2Feat-bench, and Multi-SWE-bench all tackle under-specification by augmenting task statements with hints derived from ground truth. We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork. Based on these results, we recommend techniques for deriving parsimonious hints. We will release our hints so the community can obtain more reliable signals from popular benchmarks.
| Comments: | Codes available at this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.22760 [cs.AI] |
| (or arXiv:2610.22760v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22760 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuanzhe Liu [view email]
[v1]
Fri, 16 Oct 2026 00:02:54 UTC (382 KB)
来源：arXiv cs.AI · arxiv.org