arXiv cs.AI· Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang, Weixiang Yu, Cheng Lu, Chonggao Sun, Tong Zhou, Dahua Lin, Hang Zhang, Shuai Bai, Dayiheng Liu·· 14 小时前AI 评分38
Co-Design Bench：面向智能体模型的多模态协作编程评测基准
Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models
AI 导读
研究者推出 Co-Design Bench，一个从 26,314 个真实用户编程会话的 37,463 条去重查询中筛选出 273 个任务的无参考基准，含多模态编辑与仓库编辑两条赛道及 2,331 条隐藏专家验收标准。
正文 · 原文
Authors:Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang, Weixiang Yu, Cheng Lu, Chonggao Sun, Tong Zhou, Dahua Lin, Hang Zhang, Shuai Bai, Dayiheng Liu
Abstract:Evaluating human-AI co-design requires testing whether coding agents can translate real users' multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Its two tracks, Multimodal Editing (ME) and Repository Editing (RE), cover localized edits with optional image, audio, video, or document inputs and coordinated repository revisions. Tasks span three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The tasks pair normalized requests and runnable pre-edit applications with 2,331 hidden expert-authored acceptance criteria. Our judge-then-refute protocol grades criteria using source and browser evidence, challenges positive verdicts with a separate LLM verifier, and audits visual and interaction claims without gold code or target screenshots. Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.
| Comments: | Project page: this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35138 [cs.AI] |
| (or arXiv:2610.35138v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35138 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xilin Wei [view email]
[v1]
Mon, 26 Oct 2026 10:10:21 UTC (3,846 KB)
来源：arXiv cs.AI · arxiv.org