arXiv cs.CL· Jialei Li, Dingxian Wang, Xiaoyan Zhao, Yimeng Bai, Xuanyi Zhu, Yang Zhang, Chao Yin, Wentao Shi, Ganesh Prabhu, Karthik Gomadam Rajagopal, Brett Levert, Darvin Yi, Andrew Rabinovich, Zhao Chen·· 12 小时前AI 评分40
SWE-Clar：编码智能体能否澄清不完整的客户需求并交付绿地软件？
SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?
AI 导读
研究者提出 SWE-Clar 基准，用于评估编码智能体在自由职业场景下澄清需求并实现代码的能力，包含 100 个基于 Upwork 真实开发任务（每项总付款至少 US$4,000）的任务。每个任务先进行编码智能体与客户智能体的多轮交互，再分别独立运行代码实现与澄清评估，后者由 512 道配参考答案的问题构成。九个前沿模型测试中，实现得分最高的 GPT-6 Astra 任务通过率也仅 67%。
正文 · 原文
Authors:Jialei Li, Dingxian Wang, Xiaoyan Zhao, Yimeng Bai, Xuanyi Zhu, Yang Zhang, Chao Yin, Wentao Shi, Ganesh Prabhu, Karthik Gomadam Rajagopal, Brett Levert, Darvin Yi, Andrew Rabinovich, Zhao Chen
Abstract:Greenfield software development in freelance settings requires coding agents to turn incomplete client requests into working software. This makes requirement clarification essential, yet clients may be unable to provide continuous feedback over long development cycles, and receiving their answers does not guarantee correct understanding of complex requirements. These challenges motivate two benchmark design choices: clarification precedes code implementation, and its effectiveness is assessed through explicit tests of requirement understanding. Guided by these choices, we introduce SWE-Clar, a benchmark grounded in real, high-value client commissions to evaluate requirement clarification and code implementation in this setting. It comprises 100 tasks based on real software development jobs on Upwork, each with at least US\$4,000 in total payments. Each task begins with multi-round interaction between a coding agent and a client agent, followed by separate runs for code implementation with acceptance testing and clarification assessment. For the latter, we construct 512 questions paired with reference answers from real client-freelancer interactions, covering both explicitly stated requirements and those inferred from the conversational context. Both runs begin from the same clarification context but remain isolated, preventing assessment questions from guiding code implementation. Experiments with nine frontier models show that even GPT-6 Astra, which has the highest observed implementation score, achieves a task pass rate of only 67%, underscoring the challenge of independent software delivery in this setting. The dataset and evaluation code are available at this https URL.
| Comments: | 29 pages, 9 figures, 11 tables. Code and data: this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.31854 [cs.SE] |
| (or arXiv:2610.31854v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31854 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jialei Li [view email]
[v1]
Thu, 22 Oct 2026 19:36:48 UTC (423 KB)
来源：arXiv cs.CL · arxiv.org