arXiv cs.AI· Marco Paina, Mark Niklas M\"uller, Niels M\"undler-Sasahara, Martin Vechev·· 13 小时前AI 评分44
PROBE 基准：前沿编程智能体发现的 bug 与维护者修复的不同
PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix
AI 导读
研究者推出 PROBE 基准，收录 20 个 GitHub 仓库 18 个月内修复的全部 2,457 个 bug，用于评估编程智能体发现 bug 的能力。前沿模型在该基准上召回率均低于 21%、精确率低于 9%，更强模型可将召回率从 8% 提升到 17%，但精确率没有改善。
正文 · 原文
Abstract:Proactive bug discovery is an important software-engineering problem. Even though coding agents are already widely deployed in this domain, their capabilities remain understudied. In particular, the challenging task of finding all bugs that human maintainers would repair across an entire repository has not been studied yet. To close this gap, we introduce PROBE, a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories, together with a method for matching candidate findings to this partial reference set. We evaluate coding agents across frontier models and find they all achieve a recall of below $21\%$ and a precision of less than $9\%$ with respect to the reference set. While stronger models increase recall substantially from $8\%$ to $17\%$, they do not improve precision. Surprisingly, despite the low precision, LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers. The practical relevance of bug discovery and the low recall of frontier models make this benchmark a valuable resource for both model users and developers.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.35987 [cs.SE] |
| (or arXiv:2610.35987v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35987 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mark Niklas Müller [view email]
[v1]
Mon, 26 Oct 2026 12:04:46 UTC (548 KB)
来源：arXiv cs.AI · arxiv.org