arXiv cs.AI· Josip Vrdoljak, Akash Singh, Bas Steunebrink, Alex Yarotsky, John Lorenzo, Achille Nicoletti, Sam Pigott·· 10 小时前AI 评分58
ExpertCrowd-SWE 基准：在陌生代码库中评测前沿编码智能体
ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases
AI 导读
研究者发布 ExpertCrowd-SWE（EC-SWE）基准，包含 100 个由工程师撰写的多文件软件工程任务，覆盖 87 个开源项目和 11 种语言，用于评测自主编码智能体在大型陌生代码库中的表现。
正文 · 原文
Abstract:ExpertCrowd-SWE (EC-SWE) is a benchmark of 100 expert-authored, multi-file software engineering tasks for evaluating autonomous coding agents in large, unfamiliar codebases. Each task is a change request written by a practising engineer against a pinned commit of a real repository, paired with a sealed reference solution and a task-specific grader whose weighted, mostly executable criteria award partial credit; must-pass blockers and pass/fail gates encode what a maintainer would refuse to merge. The tasks span 87 open-source projects and 11 languages. The median ticket is 148 words, while the median reference solution changes 471 lines. Every task image is calibrated before release so that the reference solution scores 1.0 and both the untouched repository and an inert change score 0; nearly half of the tasks also ship alternative, lazy and adversarial control patches that the grader must classify correctly. The final 100 tasks were selected for difficulty against current frontier models. We evaluate seven frontier models, five runs each (3,495 scored runs), in a single platform harness that gives each model its provider-native tool interface. Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) lead without a statistically significant gap, followed by Claude Fable 5.1 (46.5%), Gemini 3.8 Flash (40.0%), and a statistically indistinguishable group of Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 (32-34%). No task is fully solved by every model, and on 80 tasks no model earns full credit on all five runs. Scores span 35 points. Depending on the model, agents take a median of 33 to 166 steps per run, with little relation to score. We release 24 tasks with their environments, graders and results, and hold out 76 to limit contamination.
| Comments: | 17 pages, 5 figures, 6 tables. Dataset: this https URL ; leaderboard: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.13246 [cs.AI] |
| (or arXiv:2611.13246v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13246 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Josip Vrdoljak [view email]
[v1]
Mon, 9 Nov 2026 21:20:57 UTC (152 KB)
来源：arXiv cs.AI · arxiv.org