arXiv cs.CL· Yuhang He, Hongyu Ge, Zihan Lin, Siyi Liu, Yongqi Zhang·· 6 天前AI 评分23
S³R：通过自稀疏推测解码实现快速且忠实的 RL rollout
S$^3$R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding
AI 导读
针对长上下文 RLVR 中注意力占自回归解码内核时间 72–76% 的问题，研究者提出自稀疏推测 rollout（S³R），用稀疏注意力提议 token、再经全注意力验证与精确推测采样生成响应，使响应组仍服从稠密策略分布。在 Qwen3 上，S³R 将长上下文数学推理的端到端训练吞吐提升至自回归 rollout 的 1.39×–1.35×，任务表现相当。
正文 · 原文
Abstract:Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization. In our profiled RLVR workloads, attention accounts for 72--76\% of autoregressive decoding kernel time. Sparse attention reduces these reads, but sampling directly from a sparse policy changes the response groups used for learning. For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members. We propose Self-Sparse Speculative Rollouts (S$^3$R). It uses sparse attention to propose tokens, then generates responses through full-attention verification and exact speculative sampling. Drafting and verification share the same policy parameter, eliminating the need to train or update a separate drafter. The resulting response groups follow the dense-policy distribution. Under S$^3$R, we characterize speedup conditions and the costs of drafting and verification. We found that dense verification still requires the complete historical KV cache. A KV-aware scheduler therefore limits KV-cache repeated offloading and restoration while maintaining rollout parallelism. % OR thrashing In a matched ablation with an aggressive sparse drafting budget, S$^3$R maintains stable training while direct sparse sampling with training-side correction becomes unstable. On long-context mathematical reasoning, S$^3$R achieves $1.39\times~1.35\times$ the end-to-end training throughput of autoregressive rollouts on Qwen3 model, with comparable task performance.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13202 [cs.LG] |
| (or arXiv:2610.13202v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13202 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhang He [view email]
[v1]
Fri, 9 Oct 2026 08:22:40 UTC (6,766 KB)
来源：arXiv cs.CL · arxiv.org