arXiv cs.AI· Mohammadreza Sediqin, Shivali Dalmia, Srinivasa Karthikeya Reddy Kovvuri, Abhishek Mukherji·· 10 天前AI 评分38
为 AI 智能体评测加上信任层：arXiv 论文提出 Agent Evaluation Trust Layer
A Trust Layer for Agent Evaluation
AI 导读
研究者提出面向 AI 智能体评测的 Trust Layer，在记录分数旁标注该结果是否可信，核查四项属性：结果是否由基准自身评分逻辑支持、通过答案是否来自可追溯计算、完成声明是否与实际相符、重复执行下是否稳定。
正文 · 原文
Abstract:Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second run. We introduce a Trust Layer for Agent Evaluation, an additive post-hoc framework that reports, beside each recorded score, whether it should be believed. It verifies four properties: whether the result is supported by the benchmark's own grading logic, whether a passing answer was earned through traceable computation, whether the agent's completion claim matches what occurred, and whether the result is stable under repeated execution. The first three use only saved artifacts; the fourth re-runs the agent. Model judgments only label evidence under majority voting; all verdicts follow deterministic rules and never modify the recorded score. Applied to five agent configurations on 108 tasks from Agents' Last Exam, every model shows passing runs with no traceable computation (at rates varying tenfold), confirmed false completion claims, and unstable results: 18-46% of tasks do not stay in one score band over five runs. Only 22.6% of recorded passes clear all four checks (95% CI 15.0-32.6, n=84). Measuring what an agent can do and verifying that it did it are different problems, and current benchmarks address only the first.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07274 [cs.AI] |
| (or arXiv:2610.07274v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07274 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mohammadreza Sediqin [view email]
[v1]
Mon, 5 Oct 2026 19:15:37 UTC (90 KB)
来源：arXiv cs.AI · arxiv.org