跳到正文
arXiv cs.AI· Yangze Liu, Zhongyi Han·· 10 小时前AI 评分60

研究:编码智能体的 harness 差异多数与重跑波动相当

What Does a Harness Buy? Tokens, Mostly

AI 导读

研究将七个模型分别跑在 Claude Code、mini-SWE-agent 和 OpenCode 三个生产 harness 上,在 SWE-bench Verified 及另外两个基准上测试并原样重跑。

正文 · 原文

View PDF HTML (experimental)

Abstract:A coding agent is a language model inside a harness: the system prompt, the tools, and the context management that let a chat model work in a repository. Most of the engineering on these agents goes into the harness, and vendors and leaderboards credit it with score gains. Earlier studies have fixed the model and swapped the harness, but have not measured whether the swap changes the pass rate more than running the same harness again. We run seven models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified and two benchmarks of other kinds, and rerun them unchanged. No harness leads consistently across models. Where the data can bound the difference between two harnesses, the bound is 2 to 5 points, and a harness swap changes about as many task outcomes as a rerun. The exceptions are harness effects in particular pairings of model and harness, whose direction changes with the model: OpenCode trails by 8 to 9 points on one model and leads by 14 on another. Cost differences between harnesses also depend on the model, from under 10% to 3 to 4 times. Which harness is better therefore has no answer apart from the model, and a harness difference should be reported next to a rerun.
Comments: 31 pages, 9 figures. v2: seven models, reruns of every model, two non-SWE benchmarks; revised conclusions on pass rate and cost. Code and data: this https URL
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
Cite as: arXiv:2610.04433 [cs.AI]
  (or arXiv:2610.04433v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2610.04433

arXiv-issued DOI via DataCite

Submission history

From: Yangze Liu [view email]
[v1] Sat, 3 Oct 2026 10:47:10 UTC (406 KB)
[v2] Wed, 18 Nov 2026 13:24:25 UTC (857 KB)

来源:arXiv cs.AI · arxiv.org