arXiv cs.AI· Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu, Ci Zhang, Changdi Yang, Yanzhi Wang, Wei Niu, Jinhui Wang, Jin Lu, Geng Yuan·· 10 天前AI 评分24
BitNest：面向内存高效 LLM 推理加速的位嵌套投机解码
BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
AI 导读
BitNest 是一种位嵌套投机解码框架，将低精度草稿模型直接嵌入高精度目标模型的权重表示中，使两者共享单一物理权重，无需额外草稿模型。该方法在多个 7B–8B 边缘友好 LLM 上实现平均 95.2% 的投机接受率，端到端解码较 FP16 自回归提速 1.48–1.61x，并将渐进精度设计扩展至 KV cache 以支持长上下文推理。
正文 · 原文
Authors:Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu, Ci Zhang, Changdi Yang, Yanzhi Wang, Wei Niu, Jinhui Wang, Jin Lu, Geng Yuan
Abstract:Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-speculative approaches reduce this overhead, yet still face trade-offs between draft quality, target quality, and storage efficiency. We propose BitNest, a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation. Instead of deriving a draft from a predefined target, BitNest first constructs a strong low-precision base and then recovers the higher-precision target through residual refinement, enabling both models to share a single physical weight representation. BitNest further extends this progressive-precision design to the KV cache for long-context inference. Across multiple 7B--8B edge-friendly LLMs and diverse workloads, BitNest achieves an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and delivers 1.48--1.61x end-to-end speedup over FP16 autoregressive decoding. On the LLaMA models supported by all representative self-speculative baselines, BitNest also achieves consistently competitive or higher decoding speedup.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.02800 [cs.AI] |
| (or arXiv:2610.02800v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02800 arXiv-issued DOI via DataCite |
Submission history
From: Chence Yang [view email]
[v1]
Fri, 2 Oct 2026 04:43:42 UTC (257 KB)
[v2]
Tue, 6 Oct 2026 15:26:50 UTC (325 KB)
来源：arXiv cs.AI · arxiv.org