arXiv cs.AI· Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli·· 14 小时前AI 评分31
为何投机解码在稀疏 MoE 模型上比预期更有效
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
AI 导读
研究在 10 个稀疏 MoE 模型（30B–1T 参数、8–512 专家）上评估投机解码，发现近期细粒度模型在低 batch size 下反而获得更高加速，打破了此前"加速比与 batch size 呈凹形关系、在中等 batch 达峰"的预测。
正文 · 原文
Abstract:Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a concave speedup-batch size relationship, with gains peaking at intermediate batch sizes due to expert saturation. We evaluate this across 10 sparse MoE models (30B--1T parameters, 8--512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size. We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics. At low batch size, MoE layers in fine-grained models account for a much smaller share of runtime than of weights; on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts. Across draft tokens, routing overlap further reduces the unique experts activated during verification relative to independence assumptions. We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4\% error, and substantially narrows the gap to real-model curves and captures the qualitative low-batch trend from synthetic routing.
| Comments: | Accepted as at NeurIPS 2026 main track conference |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38981 [cs.LG] |
| (or arXiv:2610.38981v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38981 arXiv-issued DOI via DataCite |
Submission history
From: Ekagra Ranjan [view email]
[v1]
Wed, 28 Oct 2026 22:04:53 UTC (3,789 KB)
来源：arXiv cs.AI · arxiv.org