arXiv cs.AI· Pujun Zheng, Zixin Shang, Shufan Jiang, Wenhui Tian, Dongsheng Zhu, Zerun Ma, Dingbo Yuan, Qi Zhang·· 6 天前AI 评分44
SWE-Bench Pro Verified：面向软件工程智能体的可靠基准
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
AI 导读
针对 SWE-Bench Pro 存在的奖励黑客与任务质量问题，研究者推出 SWE-Bench Pro Verified。该版本通过防黑客机制消除主要泄漏渠道，并对缺陷实例做最小化修正。评估显示部分模型表现明显低于此前结果，表明原有分数可能高估了真实软件工程能力。
正文 · 原文
Abstract:SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously evaluated, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2609.08149 [cs.AI] |
| (or arXiv:2609.08149v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.08149 arXiv-issued DOI via DataCite |
Submission history
From: Dingbo Yuan [view email]
[v1]
Tue, 8 Sep 2026 02:29:58 UTC (1,396 KB)
[v2]
Wed, 16 Sep 2026 03:28:37 UTC (1,396 KB)
[v3]
Fri, 9 Oct 2026 02:16:00 UTC (1,395 KB)
来源：arXiv cs.AI · arxiv.org