KVBuffer：面向线性注意力的 IO 感知推理服务机制

View PDF HTML (experimental)

Abstract:Linear attention has recently gained significant attention for long-context inference due to its constant decoding cost with respect to context length. However, existing serving systems typically serve linear attention by recurrently computing and updating a large linear attention state in every decoding step. Since the state is much larger than the per-token key and value, recurrent decoding incurs substantial memory access and becomes inefficient for serving linear attention. In this paper, we propose KVBuffer, an IO-aware serving mechanism for linear attention. By buffering recent keys and values, KVBuffer enables serving systems to compute linear attention outputs in more flexible and memory-efficient ways. For decoding, KVBuffer enables chunkwise computation, which reduces average memory access and decoding latency by deferring state updates and applying them in batch. For speculative decoding, KVBuffer verifies draft tokens in parallel and avoids storing temporary states. For short contexts, KVBuffer computes attention outputs directly from buffered keys and values, without creating or updating the linear attention state. We implement KVBuffer in SGLang for Qwen3.5 and Nemotron-3 series models. Our evaluations show that KVBuffer can speedup batch decoding by up to 1.47x and increase the maximum number of serving requests by 5x for speculative decoding when verifying four draft tokens.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2605.19049 [cs.LG]
	(or arXiv:2605.19049v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.19049 arXiv-issued DOI via DataCite

Submission history

From: Longwei Zou [view email]
[v1] Mon, 18 May 2026 19:14:52 UTC (130 KB)
[v2] Thu, 5 Nov 2026 15:09:10 UTC (243 KB)