arXiv cs.AI· Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang, Jingmin Zhu, Hailong Yang·· 3 天前AI 评分23
SpecFit：卸载式 MoE 推理中并非所有草稿 token 代价相同
SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
AI 导读
SpecFit 通过选择预测吞吐量最高的已验证草稿子树，而非追求最长接受长度，来优化卸载式 MoE 推理中的推测解码。该方法用草稿路径概率估算提交 token 数，并根据子树激活专家的逐层并集预测周期时间，该并集为单调子模覆盖函数。
正文 · 原文
Abstract:Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15417 [cs.AI] |
| (or arXiv:2610.15417v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15417 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiuyu Li [view email]
[v1]
Sun, 11 Oct 2026 07:41:12 UTC (55 KB)
来源：arXiv cs.AI · arxiv.org