arXiv cs.CL· Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song, Xudong Ma, Hongxin Li, Yaxin Du, Weicheng Gu, Fanglin Xu, Yi Xing, XiangLong Liu, Jujie He, Zhange Zhang, Huatong Song, Zhaoyi Li, Deyang Kong, Chuan Hao, Peihao Wu, Mingjie Tang, Ran Tao, Jian Yang, Wayne Xin Zhao, Bryan Dai·· 12 小时前AI 评分38
RealCLI：面向 CLI 智能体人机协作的统一评测基准
RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
AI 导读
研究者提出 REALCLI，一个成对评测 CLI 智能体任务执行与人机协作质量的基准，包含五维框架、22 个子维度和 51 个交互场景模板。它提供 REALCLI-EXEC（一次性给出完整规格）与 REALCLI-COLLAB（脚本化用户逐步提供信息、纠正与授权）两种模式。
正文 · 原文
Authors:Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song, Xudong Ma, Hongxin Li, Yaxin Du, Weicheng Gu, Fanglin Xu, Yi Xing, XiangLong Liu, Jujie He, Zhange Zhang, Huatong Song, Zhaoyi Li, Deyang Kong, Chuan Hao, Peihao Wu, Mingjie Tang, Ran Tao, Jian Yang, Wayne Xin Zhao, Bryan Dai
Abstract:Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions. Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score ($J_{\mathrm{collab}}$), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management. REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22507 [cs.CL] |
| (or arXiv:2610.22507v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22507 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jian Yang [view email]
[v1]
Thu, 15 Oct 2026 18:03:13 UTC (1,243 KB)
来源：arXiv cs.CL · arxiv.org