arXiv cs.AI· Yirui Liu, Ruoling Qi, Xuaner Wu, Penghang Liu, Jian Chen·· 2 天前AI 评分32
Tail-Replay：突破混合 LLM 前缀缓存中线性注意力的限制
Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs
AI 导读
Tail-Replay 是一种面向混合大语言模型的前缀缓存机制，通过只缓存全注意力 KV cache、省略循环状态检查点，在缓存命中时重放匹配前缀的短近期后缀来重建线性注意力状态，从而实现无约束的 token 级前缀复用。
正文 · 原文
Abstract:Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structure complicates prefix caching: full-attention key-value caches are token-addressable, whereas linear-attention layers maintain recurrent states that cannot be rolled back to arbitrary prefix boundaries. Existing hybrid prefix caching methods address this mismatch by storing recurrent-state checkpoints. As a result, token-level matches are directly usable only at positions aligned with stored checkpoints, constraining prefix reuse to a discrete set of boundaries. We present Tail-Replay, a prefix caching mechanism that enables unconstrained token-level prefix reuse in hybrid large language models. The key insight is that linear-attention mechanisms such as Gated DeltaNet can be viewed as a structured, lossy compression of the input prefix: gated recurrent updates progressively attenuate the contributions of earlier inputs. Consequently, the recurrent state of a matched prefix can be well approximated by replaying only a short, recent suffix of that prefix. Tail-Replay exploits this property by caching the exact full-attention key-value cache while omitting recurrent-state checkpoints. On a cache hit, it reconstructs the linear-attention states by replaying a short, recent suffix of the matched prefix. As a result, the reuse boundary is determined by the shared tokens rather than by recurrent-state checkpoints. We evaluate Tail-Replay on three Gated DeltaNet-based hybrid models using the LongBench and RULER benchmarks. With only a 5--10\% replay budget, it retains 92.8--99.9\% of full-prefill quality on LongBench and RULER. For serving efficiency, we evaluate time-to-first-token speedups across multiple matched-prefix lengths---8K, 16K, and 32K. The speedup grows with prefix length, reaching $9.1$--$14.3\times$ over full prefill at 32K.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.30310 [cs.LG] |
| (or arXiv:2608.30310v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.30310 arXiv-issued DOI via DataCite |
Submission history
From: Yirui Liu [view email]
[v1]
Mon, 31 Aug 2026 06:27:07 UTC (228 KB)
[v2]
Thu, 15 Oct 2026 14:12:48 UTC (228 KB)
来源：arXiv cs.AI · arxiv.org