arXiv cs.CL· Jungseob Lee, Hogun Seong, Euijun Choi, Sugyeong Eo·· 13 小时前AI 评分31
目标模型重写数据会缩小投机解码中草稿训练目标的增益
Target Regeneration Shrinks the Gains of Draft-Training Objectives in Speculative Decoding
AI 导读
研究发现，投机解码中蒸馏等草稿训练目标带来的接受率增益，在目标模型对数据做贪婪重生成后大幅缩水：蒸馏增益变小，验证门控目标几乎无增益。该现象在视觉语言与纯文本任务、不同数据顺序、位置权重、验证器、解码温度及自回归草稿模型上均成立。作者建议目标增益应以目标重生成基线衡量，并跨训练长度评估。
正文 · 原文
Abstract:Speculative decoding accelerates large language models by verifying drafted tokens in parallel. New draft-training objectives are usually credited with the acceptance they add over cross-entropy on externally written responses, yet many pipelines let the target rewrite them. We ask how much of that gain survives. With a vision-language target and a block drafter, we train three objectives on four versions of one corpus: cross-entropy on corpus tokens, token-level distillation from the target, and distillation gated to positions that would survive verification. Distillation and the gate add large gains on external responses, but after greedy regeneration, distillation adds a smaller gain and the gate little or none. The shrinkage persists across data orders, position weights, verifiers, decoding temperatures and, for distillation, initializations and learning rates. Most of it remains at matched supervised length and at each objective's best measured training length. A second vision corpus shows the shrinkage, as does a text-only replication, for distillation also with an autoregressive drafter. Objective gains should therefore be measured against a target-regenerated baseline and across training lengths. Our code is available at this https URL.
| Comments: | 19 pages, 6 figures, 15 tables. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.29013 [cs.CL] |
| (or arXiv:2610.29013v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29013 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sugyeong Eo [view email]
[v1]
Wed, 21 Oct 2026 00:26:37 UTC (221 KB)
来源：arXiv cs.CL · arxiv.org