arXiv cs.AI· Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao, Sicheng Zhou·· 12 小时前AI 评分49
TraceTrial-Bench：面向 AI 智能体框架可靠性与故障归因的基准测试
Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
AI 导读
研究者推出 TraceTrial-Bench，将智能体框架（harness）可靠性拆为上下文、执行、治理、生命周期、可观测性、工具、验证七个组件，构建 95 个专家级任务并配套轨迹追踪与双层评估框架。
正文 · 原文
Authors:Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao, Sicheng Zhou
Abstract:LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32285 [cs.AI] |
| (or arXiv:2610.32285v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32285 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hengjie Zhu [view email]
[v1]
Fri, 23 Oct 2026 07:43:26 UTC (8,717 KB)
来源：arXiv cs.AI · arxiv.org