arXiv cs.AI· Ben Johnson, Hayden Helm·· 12 小时前AI 评分22
利用缓存轨迹实现任务高效的智能体评测
Task-Efficient Agent Evaluation Using Cached Traces
AI 导读
研究者提出用 LLM 提取执行轨迹的结构化摘要并经标准文本编码器嵌入，以构建参考系统的低维表示并预测新系统得分。在 SWE-bench Verified 和 Terminal-Bench 2.0 上，仅给定单个随机执行任务时，该方法将 item-response 模型的平均绝对误差分别从 0.13 降至 0.08、从 0.14 降至 0.12。
正文 · 原文
Abstract:Efficient benchmarking estimates a system's full benchmark performance from a small number of task executions. Standard approaches rely solely on the binary correctness of executions from a collection of already-evaluated reference systems. An agentic system, however, produces a rich trace for each execution that can reveal substantially more about its capabilities. Leveraging this information requires useful trace representations. We propose representing a trace by extracting a structured summary of it with an LLM and embedding the summary with a standard text encoder. These representations preserve correctness- and task-related signal while minimizing the presence of system identity-, model family- and harness-related signals that dominate raw trace embeddings. Given this representation for each task execution from all reference systems, we construct low-dimensional representations of the systems and predict a new system's score from its position among the reference systems. On SWE-bench Verified and Terminal-Bench 2.0, augmenting an item-response model with this geometry-based prediction reduces mean absolute error from 0.13 to 0.08 (Wilcoxon $p < 10^{-4}$) and from 0.14 to 0.12 (Wilcoxon $p < 10^{-3}$), respectively, when provided a single random executed task. Further, given access to the full set of reference systems, our approach dominates baselines for any given number of executed tasks on both benchmarks.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06803 [cs.AI] |
| (or arXiv:2611.06803v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06803 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ben Johnson [view email]
[v1]
Wed, 4 Nov 2026 17:20:09 UTC (135 KB)
来源：arXiv cs.AI · arxiv.org