arXiv cs.CL· Alagappan Valliappan·· 11 小时前AI 评分38
Windowed-MTP：在百万级 Token 上下文下消除全上下文草稿 KV 开销
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
AI 导读
Windowed-MTP 通过仅对草稿模型施加 StreamingLLM 式窗口加注意力汇、保持目标模型全上下文验证，在百万级 Token 上下文下实现无需额外接受率的 1.28-1.44x 单步解码加速。
正文 · 原文
Abstract:Speculative decoding accelerates autoregressive generation by having a draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP) draft head, assumed to be cheap beside verification. A shipped full-attention MTP head can become disproportionately expensive beside a hybrid target at long context. The head attends over the full KV cache for each of $\gamma$ draft forwards per decode step, while only 9-25% of target layers retain full attention, so a single draft layer can issue context reads at a rate comparable to the whole target. On hard, low-acceptance workloads, deep native MTP becomes net-negative: slower than no speculation.
We apply a StreamingLLM-style window plus attention sink to the draft only (Windowed-MTP); target verification remains full-context, so the target distribution is preserved. The intervention is training-free and drop-in. On three models from two hybrid families (Gated DeltaNet and Mamba2) at 1M context on one GPU in SGLang, it provides an acceptance-free 1.28-1.44x per-decode-step speedup over native MTP. The speedup grows with context; at matched acceptance, TPOT improves by the same factor. The window removes most of the draft's working set, and a compact ring buffer converts that reduction into physical capacity: it reclaims the otherwise dead draft pool, a measured 7.7-11.1% of total KV at 1M, making room for one additional concurrent 1M request in the Qwen-35B batch sweep.
| Comments: | 29 pages, 5 figures, 16 tables. v2: Windowed draft attention is merged upstream in SGLang (PR #32673). Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Performance (cs.PF) |
| ACM classes: | I.2.7; I.2.6; C.4 |
| Cite as: | arXiv:2607.21535 [cs.LG] |
| (or arXiv:2607.21535v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.21535 arXiv-issued DOI via DataCite |
Submission history
From: Alagappan Valliappan [view email]
[v1]
Thu, 23 Jul 2026 17:21:44 UTC (830 KB)
[v2]
Fri, 30 Oct 2026 17:48:26 UTC (810 KB)
来源：arXiv cs.CL · arxiv.org