arXiv cs.CL· Jungseob Lee, Giyoon Jang, Chailin Nam, Jiwoo Kum, Sihyeon Lee, Kyojun Choo, Jihun Song, Eunseo Song, Chanjun Park·· 13 小时前AI 评分22
理解投机解码中的目标概率复用
Understanding Target Probability Reuse in Speculative Decoding
AI 导读
研究显示，在投机解码中，用当前上下文单独计算的目标分布对首 token 分支重排序，可提升期望接受率，小树预算下尤为明显。先从该分布抽取首 token 再分配树，即使平均树大小相同，也比剪枝预选树更优。当单独的目标前向与独立草稿生成重叠时，两种选择规则的解码速度均快于草稿排序。
正文 · 原文
Abstract:Speculative decoding uses a target model to verify a tree of draft tokens in parallel. We study whether the resulting next-token distributions can help select the next tree. For unrestricted target models, we show that guaranteed exact reuse requires a distribution computed at the next round's complete context. We characterize when retaining this distribution requires sacrificing accepted tokens and establish conditions under which this loss persists across decoding rounds. Across six targets, we find that reranking first-token branches with a separately computed distribution at the current context improves expected acceptance over draft ranking, especially with small tree budgets. Drawing the first token from this distribution before allocating the tree further improves expected acceptance over pruning a preselected tree, even at equal mean tree size. In our experiments, both selection rules also decode faster than draft ranking when the separate target pass overlaps standalone drafting. Our findings show that making target probabilities available before tree allocation can improve the efficiency of speculative decoding. Our code is available at this https URL.
| Comments: | 29 pages, 6 figures, 29 tables. Code: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.29210 [cs.CL] |
| (or arXiv:2610.29210v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29210 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chanjun Park [view email]
[v1]
Wed, 21 Oct 2026 05:02:50 UTC (467 KB)
来源：arXiv cs.CL · arxiv.org