arXiv cs.MA· Yuxiang Lai, Jike Zhong, Chengzhi Liu, Ming Li, Haonan Wang, Xiaofeng Yang, Liang Zhao, Youbao Tang, Ruixi Fan·· 3 天前AI 评分38
CLASHarness：用可控可验证生成管线评测计算机使用中的多智能体冲突
CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline
AI 导读
研究者推出 CLASHarness（Conflict-Loaded Agent Scenario Harness），一个可控生成计算机使用场景中可验证多智能体冲突的框架，并附带 600 个多轮任务基准。
正文 · 原文
Abstract:Vision-language models (VLMs) are increasingly deployed in multi-agent systems to automate complex computer-use workflows, yet existing benchmarks mostly assume consistent, cooperative environments. In contrast, real-world collaboration rarely runs on consistent information: peer or sub-agents frequently report outdated states, issue contradictory instructions, or compete over shared permissions and files, forcing agents to check others' outputs and verify ground truth in the environment. Evaluating such conflict resolution remains challenging: manual authoring cannot scale across complex systems, while unconstrained LLM generation produces narrative drift, answer leakage, and unsatisfiable rubrics. We introduce CLASHarness (Conflict-Loaded Agent Scenario Harness), a controllable framework for generating verifiable multi-agent conflicts in computer use, with a benchmark of 600 multi-round tasks. By combining hierarchical scenario planning with an eight-step verified micro-pipeline, CLASHarness pre-commits decisive visual facts before rendering and compiles hybrid code and semantic graders directly from executed reference trajectories, guaranteeing every task is solvable by construction. Evaluating 15 frontier models on CLASHarness reveals three core bottlenecks. First, when sub-agent outputs conflict with the environment, weaker models blindly follow misleading information, whereas stronger models inspect visual evidence to verify ground truth. Second, resolving conflicts is much harder than finding them: while agents can spot inconsistencies, actively fixing system state drops task success significantly. Third, coordination protocols remain a universal bottleneck: even frontier models fail over 58% of closing actions, often forgetting to release shared locks or hand off tasks. CLASHarness provides a scalable, verifiable foundation for building reliable multi-agent systems.
| Subjects: | Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2610.19128 [cs.MA] |
| (or arXiv:2610.19128v1 [cs.MA] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19128 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuxiang Lai [view email]
[v1]
Tue, 13 Oct 2026 20:01:18 UTC (6,538 KB)
来源：arXiv cs.MA · arxiv.org