arXiv cs.CL· Jongwoo Ko, Pashmina Cameron·· 11 小时前AI 评分31
HyFlash：面向端侧投机解码的前缀记忆先验
HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding
AI 导读
HyFlash 是一种紧凑的并行草稿模型，通过前缀记忆先验将长程上下文建模与细粒度草稿位置交互分离，在有限容量下实现端侧投机解码。它用局部注意力捕捉近期上下文与草稿位置交互，并用循环记忆将历史目标特征压缩为固定大小状态，从而以有界草稿侧内存完成局部与长程建模。实验显示其接受率高于全上下文注意力，KV 存储显著更少，且对浅层草稿模型和长上下文增益更大。
正文 · 原文
Abstract:Parallel block drafting accelerates speculative decoding by predicting multiple future positions in a single backbone pass, but on-device deployment constrains both drafter capacity and context storage. Under tight memory budgets, drafters must model long-range dependencies and fine-grained draft-position interactions without the growing KV-cache overhead of full-context attention. We propose HyFlash, a compact parallel drafter that introduces a prefix-memory prior to separate long-range context modeling from fine-grained draft-position interactions with limited capacity. Local attention captures recent context and draft-position interactions, while recurrent memory compresses historical target features into fixed-size states. This inductive bias enables effective local and long-range modeling with bounded draft-side memory. Experiments show that HyFlash achieves higher acceptance than full-context attention with substantially less KV storage and with larger gains for shallow drafters and long contexts.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.07263 [cs.CL] |
| (or arXiv:2611.07263v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07263 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jongwoo Ko [view email]
[v1]
Thu, 5 Nov 2026 00:31:21 UTC (144 KB)
来源：arXiv cs.CL · arxiv.org