arXiv cs.CL· Han Peng, Xing Hu, Feifei Niu, Huan Li, Xin Xia·· 6 天前AI 评分29
SWE-bench 补丁性能实证研究：正确修复的运行表现差异
Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution
AI 导读
一项实证研究利用 SWE-bench 和 SWE-bench-Live 的内置 FAIL_TO_PASS 测试评估补丁运行性能，选取 Claude Opus 5、GPT 5.6、DeepSeek V4 Pro 和 Qwen 3.8 Max 四个前沿模型搭配 SWE-agent，在 262 个存在至少两种正确解法的案例上测量指令数与墙钟时间。
正文 · 原文
Abstract:Repository-level issue-resolution benchmarks such as SWE-bench evaluate submitted patches primarily based on whether they pass the correctness tests, leaving the runtime performance of different test-passing patches largely unexplored. Evaluating patch performance is essential but challenging since designing performance workloads for each benchmark instance is costly. However, these benchmarks already provide a natural candidate: the built-in \emph{FAIL\_TO\_PASS} (F2P) tests, which exercise the behavior affected by the issue fix. We therefore investigate whether the provided F2P tests can be repurposed to evaluate patch performance. Overall, we choose four frontier models, Claude Opus~5, GPT~5.6, DeepSeek V4 Pro, and Qwen~3.8 Max, paired with SWE-agent to resolve the pre-filtered measurable corpus from verified splits of SWE-bench and SWE-bench-Live, yielding 262 cases with at least two different correct solutions, including the developer ground-truth patch. We then measure each solution's runtime performance in instruction count and wall-clock time on the native F2P test, and use Welch ANOVA instead of the RSD proposed in prior work as a more robust statistical discrimination criterion. We find that this methodology exhibits notable discrimination capability in runtime performance and identifies performance differences among patches in almost half of the instances. Our empirical study shows that the runtime performance of generated patches is independent of the issue resolution rate of LLMs, with nearly reversed model rankings. The case study also shows that the performance analysis reveals architectural regression in some patches that the correctness evaluation overlooked. These findings suggest that resolution rate does not adequately characterize LLMs' capability for repository-level issue resolution, motivating future evaluations to consider patch performance alongside it.
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13563 [cs.SE] |
| (or arXiv:2610.13563v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13563 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Han Peng [view email]
[v1]
Fri, 9 Oct 2026 13:17:53 UTC (795 KB)
来源：arXiv cs.CL · arxiv.org