arXiv cs.AI· Yibo Wang, Yuanxin Liu, Zhaoran Wang·· 3 天前AI 评分24
VSR：面向可验证奖励强化学习的方差最优状态重采样
Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
AI 导读
针对可验证奖励强化学习中大量 rollout 预算浪费在无学习信号状态上的问题，研究者提出方差驱动的状态重采样方法 VSR，其闭式解为 Neyman 分配 μ_var^π(s) ∝ d^π(s)·σ_π(s)，并自然关联 GRPO 的组相对优势、随策略改进涌现出反向课程。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, $\mu_{\mathrm{var}}^\pi(s) \propto d^\pi(s)\,\sigma_\pi(s)$, which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15888 [cs.LG] |
| (or arXiv:2610.15888v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15888 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yibo Wang [view email]
[v1]
Sun, 11 Oct 2026 20:09:24 UTC (6,888 KB)
来源：arXiv cs.AI · arxiv.org