arXiv cs.AI· Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruizi Wang, Zhouyi Li, Khanh Chi Le, Young-Jun Lee, Dongyeop Kang·· 2 天前AI 评分32
Kolab：面向长周期工作流的人机协作基准测试
Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows
AI 导读
研究者推出 Kolab，一个包含 50 位来自 40 种职业的专业人士、115+ 小时屏幕录制的人机协作数据集，用于评测长周期真实工作流中的协作表现。该基准要求智能体预测专家会采纳、修改还是拒绝其贡献，并判断协作满意度与产出质量。结果显示，前沿模型（无论是否具备智能体能力）都会高估与专家的协作效果，且难以区分高质量与低质量协作，更长的上下文并未稳定改善预测。
正文 · 原文
Abstract:When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on such benchmarks does not imply effective real-world collaboration. Professional workflows offer a useful setting, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 115+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long-horizon problems. Kolab also contains participants' collaboration and output quality judgments. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback. Our benchmark operationalizes both: (1) forecasting how professionals use agent contributions at diverse workflows, and (2) verifying their judgments of collaboration satisfaction and output quality. Findings show that frontier models, both agentic and non-agentic, overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. Longer workflow context does not consistently improve these predictions. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for studying human-agent collaboration grounded in real professional workflows.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.21109 [cs.AI] |
| (or arXiv:2610.21109v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21109 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shirley Anugrah Hayati [view email]
[v1]
Thu, 15 Oct 2026 02:13:36 UTC (7,970 KB)
来源：arXiv cs.AI · arxiv.org