SWE-Prometheus：衡量真实代码仓库工程治理改进的基准
SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories
SWE-Prometheus 是一个面向仓库工程治理改进的基准，包含 60 个仓库，要求智能体在固定快照下自主识别风险、排定干预优先级并验证改动。在共享的 22 仓库公开子集上，十个模型的平均 NGI 从 0.0568 到 0.5760，行为破坏率为 0% 至 23%；冻结的十仓库批次上，仓库盲模板平均 NGI 为 0.272，但在可复现环境与依赖安全维度上无一改进。
Authors:Jiajun Wu, Leixin Sun, Zihan Tan, Yitao Liu, Zhenjun Zhao, Danyang Zhang, Shuo Li, Jiaru Qian, Yuxin Wu, Shanghaoran Quan, Chuangxin Zhao, Yangxu Liao, Yang Liu, Bin Chong, Guancheng Wan
Abstract:Large language model based coding agents have made substantial progress on repository-level software engineering tasks. Existing repository benchmarks, however, usually start from a human-identified issue and evaluate whether a patch satisfies a functional signal. We present SWE-Prometheus, a benchmark for the broader task of improving repository engineering governance. Each task provides a fixed snapshot and an open-ended objective, requiring the agent to identify risks, prioritize interventions, and verify the resulting changes. SWE-Prometheus evaluates six governance dimensions through paired evidence, clean-environment probes, behavior gates, and two independent teacher ratings of the same evidence. The benchmark contains 60 repositories; ten models are evaluated on a shared 22-repository public subset, where mean Normalized Governance Improvement ranges from 0.0568 to 0.5760 and observed behavior-breakage rates range from 0% to 23%. On a frozen ten-repository batch, a repository-blind template obtains mean NGI 0.272, but its gains concentrate in Tests & CI, Quality Gates, and Documentation; it improves Reproducible Environment and Dependency & Security on none of the repositories. This baseline makes the distinction between adding governance artifacts and producing execution-backed improvements measurable. The no-op condition has median NGI zero and standard deviation 0.073; two teachers agree exactly on 57 of 60 dimension scores for the same no-op evidence. For the two highest conditional-mean systems, common-valid NGI is similar, while full-pool comparisons that include behavior failures favor Kimi-K3. These results show why repository-governance evaluation should report improvement, behavior preservation, evidence quality, and coverage together.
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2609.29465 [cs.AI] |
| (or arXiv:2609.29465v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.29465 arXiv-issued DOI via DataCite |
Submission history
From: Jiajun Wu [view email]
[v1]
Thu, 24 Sep 2026 12:23:52 UTC (4,283 KB)
[v2]
Sat, 3 Oct 2026 18:48:42 UTC (4,291 KB)
[v3]
Thu, 15 Oct 2026 08:27:31 UTC (3,354 KB)
来源：arXiv cs.AI · arxiv.org