arXiv cs.CL· Bangji Yang, Hongbo Ma, Ruihan Guo, Jiajun Fan, Xi Zhu, Ge Liu·· 12 小时前AI 评分34
Batched Contextual Reinforcement II：RLVR 是否需要长 rollout 窗口？
Batched Contextual Reinforcement II: Does RLVR Need Long Rollout Windows?
AI 导读
作为 BCR 的技术续作，CCR 将三个可独立验证的问题放入一次受限训练补全中，让截断前完成的正确答案保留奖励，从而在短窗口下维持 RLVR 的正确性信号。
正文 · 原文
Abstract:Long rollout windows make reinforcement learning with verifiable rewards (RLVR) expensive, yet simply shortening them can remove the correctness signal: when every single-problem rollout truncates before an answer, group-relative accuracy advantages vanish. As a technical continuation of Batched Contextual Reinforcement (BCR), we study how concurrent rewards relax this dependence on long windows. Concurrent Constraint Reinforcement (CCR) places three independently verifiable problems in one capped training completion. Correct answers completed before truncation retain credit, while a shared budget encourages shorter useful reasoning without an explicit length penalty. We connect this mechanism to conditional reward- and update-variance analyses and study 4B and 8B mathematical reasoning models. Against single-problem GRPO with a 32k window, CCR with an 8,000-token window reduces tokens per training step from 975,464 to 174,957 on 4B and from 793,832 to 165,788 on 8B, saving 82.1% and 79.1%, respectively. An 8B training trace shows increasing accuracy reward alongside shorter completions. Single-problem evaluations provide supporting evidence of retained reasoning quality. These results identify reward-preserving training structure as a practical route to lower per-step cost in short-window RLVR.
| Comments: | 22 pages, 3 figures, 2 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22887 [cs.LG] |
| (or arXiv:2610.22887v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22887 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongbo Ma [view email]
[v1]
Fri, 16 Oct 2026 03:40:48 UTC (220 KB)
来源：arXiv cs.CL · arxiv.org