arXiv cs.AI· Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu·· 6 天前AI 评分41
AgentSWE：编码智能体能否构建你真正想要的智能体？
AgentSWE: Can Coding Agents Build the Agent You Actually Want?
AI 导读
研究者推出 AgentSWE 基准，覆盖智能体开发全生命周期的创建、编辑与优化三个阶段，共 25 项任务，每项以"委托"而非"考试"形式呈现。评测多种模型与框架下的编码智能体后发现，无一能接近需求要求，演示效果与最终交付之间的差距是常态而非个例。失败轨迹分析还识别出对可见案例过拟合、代码看似完整但运行时空洞、无依据宣称完成等反复出现的失败模式。
正文 · 原文
Abstract:As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.
| Comments: | Code: this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.13361 [cs.SE] |
| (or arXiv:2610.13361v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13361 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiahao Wang [view email]
[v1]
Fri, 9 Oct 2026 10:35:27 UTC (887 KB)
来源：arXiv cs.AI · arxiv.org