arXiv cs.AI· Matej Gerek, Danylo Zahorulko, Samuel Budai, Chris Rupley, Thierry Donneau-Golencer·· 10 小时前AI 评分38
基于仓库的评分标准:面向编码智能体的免执行评估方法
Repository-Grounded Rubrics for Execution-Free Evaluation of Coding Agents
AI 导读
研究将原本用于候选补丁筛选的 agentic rubrics 改造为免执行评估指标:智能体在改动前探索仓库并写出加权二元标准,无工具 judge 按满足标准的加权比例给 diff 打分。
正文 · 原文
Abstract:Coding agents are usually evaluated by running task-specific tests, but tests may be missing or incomplete, and running them requires a working environment for each repository. We adapt agentic rubrics, introduced for selecting among candidate patches, into an execution-free measure of how far a patch satisfies a task's requirements. An agent explores the repository before the change and writes weighted binary criteria, and a judge without tools scores a diff by the weighted share of criteria it satisfies. We validate the ordering this score produces against test outcomes on SWE-bench Verified, using 7,291 labeled patches from fifteen systems. Within the 400 tasks that have both resolved and unresolved patches, the mean per-task AUROC is 0.826 (0.838 on tasks not used to tune the rubric-generation prompt), and ranking systems by their failing patches closely agrees with ranking them by resolve rate (Spearman 0.968). Case analyses trace disagreements to tests that do not match the task, execution-blocking defects that the textual assessment misses, and errors in criteria or verdicts. The validation does not establish absolute score magnitudes or requirement satisfaction beyond agreement with tests, but rubric scores could complement tests where tests or execution environments cannot be assumed.
| Comments: | 31 pages, 6 figures, 15 tables. Code and data: this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23309 [cs.SE] |
| (or arXiv:2611.23309v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23309 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Matej Gerek [view email]
[v1]
Wed, 11 Nov 2026 09:21:43 UTC (125 KB)
来源:arXiv cs.AI · arxiv.org