arXiv cs.AI· Jiahong Dai, Zhuochen Yang, Pengyang Shao, Kelvin Ng, Zhongyi Liu, Chengquan Ju, Yuting He, Bo Hu·· 10 小时前AI 评分42
Hearsay:谁能审计已部署 Agent Harness 写下的运行记录?
Hearsay: Who Can Audit the Record a Deployed Agent Harness Writes?
AI 导读
研究者提出 Hearsay 基准,检验审计者能否核查 Agent 运行记录:在全部 16 个框架中,harness 都是唯一记录写入者,此时读者最多能在十次运行中的九次从记录指出故障,却无法验证记录本身。
正文 · 原文
Abstract:The record of an agent's run is written by the harness that drives the agent. When the run is later disputed, the auditor was not there and has only this record. We ask whether someone who was not there can check the record, and what decides the answer. It depends on who wrote what a check can rely on: if the harness wrote all of it, the record is only the harness's own word, and we call it hearsay. We build Hearsay, a benchmark that examines the record and not the task: five harnesses run fourteen tasks, model examiners from three vendors and a human panel read the records and answer questions about the runs, and every passage they cite is labelled mechanically by who wrote it. We also audit what is deployed: in all sixteen frameworks, the harness is the only writer. With one writer, readers can name the fault from the record, in up to nine runs in ten, but they cannot check the record. The model examiners claim corroboration for 18 to 55 points more of their evidence than the labels support, a gap we then predicted for runs driven by another vendor's model and found again. And the best of them notices only half of the alterations we make to a record, while the two that name the fault most often cannot tell what a harness left out. With a second writer, the record can be checked. Our sandbox keeps a ledger of what passes between harness and model, and reading the ledger against the record in both directions shows what the record leaves out and what it adds. We test this as an auditor would use it, on one unseen record at a time. On runs driven by a third vendor's model, with every rule committed before the records existed, it reports 64 of 66 alterations and flags none of 84 clean records. On held-out records in five formats it reports 109 of 115. What makes a record evidence is a second writer, not a better reader.
| Comments: | 50 pages (10-page main text), 8 figures. Under review. v2: re-organized around one question; the second writer is now evaluated one unseen record at a time, on held-out records and under rules committed before the records existed; title changed. Code and data: this https URL |
| Subjects: | Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE); Software Engineering (cs.SE) |
| Cite as: | arXiv:2609.32495 [cs.CR] |
| (or arXiv:2609.32495v2 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2609.32495 arXiv-issued DOI via DataCite |
Submission history
From: Jiahong Dai [view email]
[v1]
Sat, 26 Sep 2026 11:37:06 UTC (264 KB)
[v2]
Wed, 18 Nov 2026 12:50:47 UTC (318 KB)
来源:arXiv cs.AI · arxiv.org