arXiv cs.AI· Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan·· 10 天前AI 评分28
CheckerBench：长时程智能体能否合成静态分析检查器？
CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?
AI 导读
研究者推出 CheckerBench，一个由 297 个 CVE 衍生出 300 个任务的可执行基准，覆盖 167 个代码仓库、85 个 CWE 和五种语言生态，用于评估智能体能否从零完成静态分析检查器的开发。
正文 · 原文
Authors:Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan
Abstract:Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection and rarely assess whether an agent can develop a working checker in a repository from start to finish. We introduce CheckerBench, an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems. Each task includes vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. We further introduce CheckerLab, a common evaluation framework that independently rebuilds submitted checkers and measures vulnerable-fixed diagnostic contrast, patch localization, false positives, and tool use. Across 21 model-harness configurations and three independent repeats per configuration, mean Pass@1 is 32.30%, while the best reaches 45.33%. These results show that reliable, reusable checker development remains challenging for current coding agents.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2610.07557 [cs.SE] |
| (or arXiv:2610.07557v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07557 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hang He [view email]
[v1]
Tue, 6 Oct 2026 00:42:45 UTC (5,453 KB)
来源：arXiv cs.AI · arxiv.org