arXiv cs.AI· Nilesh Prasad Pandey, Jangseon Park, Jason Kong, Onat Gungor, Tajana Rosing·· 12 小时前AI 评分31
NgramRL：用面向 rollout 的 N-gram 推测解码加速结构化生成的 RL 后训练
NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding
AI 导读
NgramRL 利用同一 prompt 近期 rollout 历史构建 per-prompt n-gram drafter，无需辅助模型或 drafter 训练，即可加速 LLM 的 RL 后训练 rollout 生成。
正文 · 原文
Abstract:Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but its cost is dominated by autoregressive rollout generation, which can exceed 70% of training time. Existing speculative-decoding methods are poorly suited to this setting: model-based drafters consume scarce GPU memory and may become misaligned as the policy evolves, while existing model-free methods do not explicitly exploit repetition across training epochs. We observe that recent rollouts of the same prompt provide a high-quality, self-updating draft source. Based on this insight, we propose NgramRL, a per-prompt n-gram drafter constructed from bounded recent rollout history. NgramRL requires no auxiliary model, drafter training, or persistent GPU-resident drafter state, while target-model verification preserves the rollout distribution. To hide CPU cache lookup and scheduling, we further introduce an adaptive fan-out prefetcher that predicts likely verification outcomes (j, b), consisting of the accepted draft length j and target-emitted continuation token b. A causal budget allocator concentrates prefetches on high-probability outcomes, while uncovered outcomes fall back to sequential lookup without changing committed tokens. Across two model families and three reasoning and structured-generation workloads, NgramRL commits 3.60-7.77 tokens per target-model forward in the warm-cache regime, up to 1.99x more than SuffixDecoding and 7.0x more than prompt-lookup decoding. On Spider, the prefetcher covers 96.6% of Qwen3-8B outcomes at a candidate budget of 16 and improves measured rollout throughput by 1.38x over synchronous NgramRL, approaching its 1.40x overlap ceiling.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32881 [cs.AI] |
| (or arXiv:2610.32881v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32881 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nilesh Prasad Pandey [view email]
[v1]
Fri, 23 Oct 2026 16:47:29 UTC (1,751 KB)
来源：arXiv cs.AI · arxiv.org