arXiv cs.AI· Toby D. Pilditch, Konstantinos Voudouris, Alexandra Abbas, Cozmin Ududec·· 10 天前AI 评分28
Transect：为长周期 LLM 智能体评估保留可观测性
Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations
AI 导读
Transect 是一个基于 Inspect Scout 的开源工具包，用于分析长周期 LLM 智能体评估中数百页的转录记录，将事件、token 用量、子智能体活动和模型生成的行为标签对齐到统一的回合时间线上。在一项生成近 1300 万 token 的 AI R&D 评估中，该工具将智能体工作划分为与研究技能分类对齐的行为阶段，发现其集中于操作性工作和稿件产出，几乎没有持续假设生成阶段的证据。
正文 · 原文
Abstract:Frontier AI evaluations increasingly use open-ended, agentic, long-horizon tasks whose transcripts can span hundreds of pages of outputs and actions from complex multi-agent networks. The observability envelop-the range of what evaluators can reliably infer about an agent's behaviours-is therefore narrowing. Language model assistants can help classify and interpret agent behaviour but also afford human evaluators significant analytical degrees of freedom, threatening the reproducibility and auditability of language-model-based transcript analysis. Transect is an open source package built on Inspect Scout to help evaluators understand how a long agent run unfolded, identify behaviour worth investigating, and check interpretations against the transcript. Users specify task context and behavioural vocabulary in a reusable evaluation-family configuration, with judge models and analysis settings supplied separately. Transect's navigable reports align recorded events, token use, sub-agent activity, and model-generated behavioural labels on a common turn-based timeline. Reviewers can quickly grasp a run's narrative, trace any label or event to its source turns, and export the underlying data tables for cross-run analysis. We demonstrate the workflow on an AI R&D evaluation that generated almost 13 million tokens, dividing the agents' work into behavioural phases aligned with research-skill classifications, sub-agent delegations and interactions, and token use. The combined view shows a focus on operational work and manuscript production, with little evidence of a sustained hypothesis generation stage-arguably a necessary component for high-quality scientific outputs. Transect's flexible, customisable transcript-analysis pipeline will enable evaluators to keep pace with longer, more complex, more frequent AI evaluations while supporting scientific rigour, transparency, and reproducibility.
| Comments: | 27 pages, 5 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.08364 [cs.AI] |
| (or arXiv:2610.08364v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08364 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Toby Pilditch [view email]
[v1]
Tue, 6 Oct 2026 13:52:15 UTC (216 KB)
来源：arXiv cs.AI · arxiv.org