arXiv cs.AI· Shao Zhang, Hangfan Zhang, Yuanyuan Shi, Hao Li, Min-Le Su, Kangcong Li, Jingping Wang, Chunjiang Mu, Chen Zhang, Yang Chen, Bo Zhang, Shuyue Hu·· 12 小时前AI 评分52
智能体失败该怪模型还是执行框架？论文提出五项指标评估模型与框架交互
When Agents Fail: Model or Harness--Who's to Blame?
AI 导读
论文提出一个评估框架，用有效完成率（VCR）、模型-框架匹配度、跨模型匹配度、交互残差和 Token 负载五项指标，衡量 LLM 智能体表现中模型与执行框架各自的贡献。
正文 · 原文
Authors:Shao Zhang, Hangfan Zhang, Yuanyuan Shi, Hao Li, Min-Le Su, Kangcong Li, Jingping Wang, Chunjiang Mu, Chen Zhang, Yang Chen, Bo Zhang, Shuyue Hu
Abstract:LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how reliably agents finish normally and deliver evaluable results, irrespective of task correctness; (ii) Model-Harness Fit, how well a harness supports task achievement through valid completion for a given model; (iii) Cross-Model Fit, a harness's overall suitability across evaluated models; (iv) Interaction Residuals, pairing-specific execution strengths and weaknesses hidden by component-level summaries; and (v) Token Load, the token demands of agent execution, including failed attempts. We evaluate four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) with four core models, plus two fixed native model-harness systems, on six benchmarks covering context management, tool use, long-context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions, 111 evaluation settings yield 52,168 trajectories. Harness rankings vary across models and benchmarks; task performance and valid completion can favor different pairings, and similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses on the shared four-benchmark panel. These findings support evaluating model-harness interactions, training transferable harness-use capabilities in models, and harness designs adapted to the model and task.
| Comments: | Preprint under review |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.32483 [cs.AI] |
| (or arXiv:2610.32483v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32483 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shao Zhang [view email]
[v1]
Fri, 23 Oct 2026 10:57:54 UTC (4,170 KB)
来源：arXiv cs.AI · arxiv.org