arXiv cs.CL· Radin Shayanfar, Keheliya Gallaba, Ahmed E. Hassan·· 13 小时前AI 评分36
如何为实践选择编码智能体基准：任务需求与智能体行为的经验教训
Choosing Coding-Agent Benchmarks for Practice: Lessons from Task Demands and Agent Behaviour
AI 导读
研究分析五个公开基准的 2,487 个任务和六种智能体配置的 14,922 次运行，提出 Spread-Novelty-Centrality（SNC）画像刻画参考解在文件与目录间的分布、增删比例及所触代码的依赖与复杂度。
正文 · 原文
Abstract:Software teams evaluating coding agents need evidence that reflects the changes they expect those agents to make. Benchmark scores summarize how often an agent resolves tasks under broad labels such as bug fixing and feature implementation, but provide little guidance on whether the evaluated work resembles a team's workload. To support this decision, we characterize 2,487 tasks from five public benchmarks and analyze 14,922 runs of six agent configurations. We introduce the Spread-Novelty-Centrality (SNC) profile to describe how the reference solution is distributed across files and directories, how much it adds relative to what it removes, and the dependencies, history, and complexity of the code it touches. We also compare the scope of agent patches and their file exploration. Every pair of benchmarks differs on at least two SNC axes, including benchmarks with the same task label. Among tasks resolved by Claude Opus 4.6, patches are larger than the reference on FeatBench and smaller on FeatureBench. Across the six configurations, resolved runs tend to involve less distributed changes and code with lower values on the Centrality indicators, while the patch sizes associated with resolution differ between the Claude and Qwen configurations. These observations suggest that practitioners should select evaluation tasks based on their change characteristics, inspect the information provided in task descriptions, and assess patch scope alongside test outcomes. We explain how teams can apply this analysis to completed changes and identify the additional evidence needed before using it for task assignment or review automation.
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.01271 [cs.SE] |
| (or arXiv:2609.01271v2 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2609.01271 arXiv-issued DOI via DataCite |
Submission history
From: Radin Shayanfar [view email]
[v1]
Tue, 1 Sep 2026 14:04:15 UTC (406 KB)
[v2]
Tue, 27 Oct 2026 02:35:36 UTC (368 KB)
来源：arXiv cs.CL · arxiv.org