arXiv cs.AI· Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen·· 2 天前AI 评分24
Hindsight-Privileged RLVR：无需组采样的单轨迹 Token 信用分配
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
AI 导读
研究者提出 HP-RLVR，一种单轨迹 RLVR 方法，利用已完成轨迹的未来信息生成 token 级更新权重，无需参考解或学习型 critic。在 Qwen3-8B 上，其数学、代码生成和 ALFWorld 训练速度分别为 GRPO 的 1.21×、1.46× 和 2.30×，每 prompt 采样量仅为后者的八分之一。
正文 · 原文
Authors:Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen
Abstract:Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared $100$-step schedule, HP-RLVR trains $1.21\times$, $1.46\times$, and $2.30\times$ faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, $43.2\%$ (4B) and $49.0\%$ (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20497 [cs.LG] |
| (or arXiv:2610.20497v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20497 arXiv-issued DOI via DataCite |
Submission history
From: Wei Chen [view email]
[v1]
Wed, 14 Oct 2026 17:06:59 UTC (176 KB)
[v2]
Thu, 15 Oct 2026 06:36:16 UTC (176 KB)
来源：arXiv cs.AI · arxiv.org