arXiv cs.AI· Markus D\"ucker, Vaibhav Kumar, Yi Liu, Ronak Chaudhary, Andreas Plesner, Francisco Guzm\'an, Anish Athalye·· 23 小时前AI 评分52
论文研究评分标准式智能体环境中的验证器,开源 BVB 基准与 Gandalf
Verifying Agents in Rubric-Graded Environments
AI 导读
论文对评分标准式(rubric-graded)智能体环境中的验证器做了系统研究,构建了覆盖 24 个投资银行任务、2,464 条人工判定评分标准的元评测数据集 BankerVerifierBench(BVB)。
正文 · 原文
Abstract:As AI agents take on increasingly open-ended and complex tasks, verifying their outputs has proven to be difficult. Rubric-graded agent environments, where a verifier judges multiple natural-language criteria against the deliverables and environment state changes produced by the agent, have emerged as a popular paradigm. We conduct the first systematic study of verifiers for rubric-graded agent environments. First, we create BankerVerifierBench (BVB), a meta-evaluation dataset of 2,464 human-judged rubric criteria across 24 investment-banking tasks. Next, we develop a methodology that derives from any rubric corpus the capabilities a verifier should possess, yielding a nine-capability taxonomy that we distill into three general design principles---reactive verification, environment alignment, and domain guidance---which we implement in Gandalf, an open-source verifier. Finally, we evaluate Gandalf against three existing verifiers on BVB. Gandalf matches or exceeds every baseline's F1 at lower cost: with Gemini 3 Flash it matches the strongest baseline's F1 (0.864 vs. 0.852) at one-fifth of its cost, and with GPT-5.4 models it improves F1 by over 0.12 at lower cost. Gandalf also generalizes to OpenClaw, a structurally different personal-productivity environment, where it again achieves the highest F1 (0.955 vs. 0.934 for the best baseline). We release BVB, Gandalf, and the code and data to reproduce our experiments.
| Comments: | Presented at NeurIPS E&D 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22841 [cs.AI] |
| (or arXiv:2611.22841v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22841 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Andreas Plesner [view email]
[v1]
Tue, 17 Nov 2026 14:21:38 UTC (59 KB)
来源:arXiv cs.AI · arxiv.org