arXiv cs.CL· Yueshen Li, Kamer Ali Yuksel, Gokhan Tur, Dilek Hakkani-T\"ur, Jo\~ao Pedro Maia·· 13 小时前AI 评分35
VeriEvolve：面向工具使用智能体的评估器优先进化框架
VeriEvolve: Evaluator-First Evolution for Tool-Using Agents
AI 导读
VeriEvolve 提出评估器优先的进化框架，先为任务合成可执行评估器再优化智能体。在 30 个 MCPMark Filesystem 任务上构建 30 个评估器与 141 个挑战状态，生成套件一致率达 141/141，留出状态为 113/122。评估器引导的进化将 GLM-5.2 的外部验证成功率从 6/30 提升至 18/30，Claude 从 27/30 提升至 30/30。
正文 · 原文
Abstract:A central obstacle to self-improving agents is the absence of a reliable fitness signal for realistic stateful tasks. We introduce \textsc{VeriEvolve}, an evaluator-first evolution framework that constructs this missing evaluation layer before optimizing the agent. From task instruction, VeriEvolve synthesizes instruction-grounded challenge states and an executable evaluator that checks them. During each round of workflow evolution, the evaluator is frozen and serves as the fitness signal. Between rounds, VeriEvolve revises the evaluator by converting the end states of strong and weak runs into a pair of counterfactual test cases.
Across all 30 MCPMark Filesystem tasks, VeriEvolve constructs 30 evaluators with 141 challenge states, achieving 141/141 agreement on the generated suites and 113/122 agreement on held-out states produced by executor agents. Evaluator-guided evolution increases externally verified success from 6/30 to 18/30 for GLM-5.2 and from 27/30 to 30/30 for Claude; refinement removes all three false positives that the weak executor exposes. Cross-environment studies on PostgreSQL, Playwright, Notion, and GitHub further expose complementary limits from observability, state representation, and search/execution capacity. These results show that executable evaluators can be synthesized and used to guide search when task success is observable and an external verifier is available for calibration.
| Comments: | 24 pages, 3 figures. Accepted to the REALM Workshop at EMNLP 2026 (non-archival) |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.29005 [cs.SE] |
| (or arXiv:2610.29005v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29005 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yueshen Li [view email]
[v1]
Wed, 21 Oct 2026 00:13:29 UTC (241 KB)
来源：arXiv cs.CL · arxiv.org