arXiv cs.AI· Huining Yuan, Zelai Xu, Huaijie Wang, Xiangmin Yi, Jiaxuan Gao, Xiao-Ping Zhang, Yu Wang, Chao Yu, Yi Wu·· 3 天前AI 评分29
VPR：面向智能体推理的可验证过程奖励框架
Verifiable Process Rewards for Agentic Reasoning
AI 导读
研究者提出可验证过程奖励（VPR）训练框架，将任务求解器改造为动作级过程验证器，为中间动作提供密集奖励，实现细粒度信用分配。VPR 在 Sokoban、Sudoku、Minesweeper 三种环境中实例化，训练的 Qwen3-4B 全面超越 outcome-reward GRPO 和 VinePPO；在数学数据与三个游戏环境联合训练时取得最高平均通用推理性能。
正文 · 原文
Abstract:Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet existing approaches predominantly rely on sparse outcome-level feedback, posing a substantial credit-assignment challenge in long-horizon agentic tasks. In this work, we study structured agentic reasoning problems that admit symbolic or algorithmic task-solving oracles. We propose Verifiable Process Rewards (VPR), a training framework that repurposes such task oracles as action-level process verifiers, providing dense rewards for intermediate actions and enabling fine-grained credit assignment. We instantiate VPR in three distinct settings: a search-based oracle for Sokoban, a constraint-based oracle for Sudoku, and a posterior-based oracle for Minesweeper. To optimize policies with these rewards, we introduce oracle-guided state-group rollout and locally normalized optimization. Empirically, VPR-trained Qwen3-4B outperforms outcome-reward GRPO and VinePPO across all three training environments. When jointly trained on math data and all three game environments, VPR achieves the highest average general-reasoning performance and strongest agentic results among the compared training configurations. To further assess its applicability to more general agentic tool-use settings, we apply VPR-style action-level supervision to $\tau^2$-bench using an expert LLM as a reference policy and observe improved performance over outcome-reward GRPO on held-out tasks. Together, these results position VPR as a promising approach for leveraging reliable task oracles to improve agentic reasoning, while highlighting its dependence on the availability and reliability of task oracles and the challenge of extending intermediate verification to less structured, open-ended environments.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.10325 [cs.AI] |
| (or arXiv:2605.10325v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10325 arXiv-issued DOI via DataCite |
Submission history
From: Huining Yuan [view email]
[v1]
Mon, 11 May 2026 10:30:53 UTC (2,152 KB)
[v2]
Wed, 27 May 2026 13:12:06 UTC (2,152 KB)
[v3]
Mon, 12 Oct 2026 08:50:38 UTC (2,506 KB)
来源：arXiv cs.AI · arxiv.org