SeKV：面向长上下文 LLM 推理的分辨率自适应 KV Cache

View PDF HTML (experimental)

Abstract:Large language models increasingly operate over long contexts, where the KV cache grows linearly with sequence length and dominates memory during decoding. Token eviction discards entries permanently, and semantic compression fixes each unit's resolution at prefill, so detail that becomes relevant later cannot be recovered. We propose SeKV, a resolution-adaptive KV cache built around a trained zoom-in mechanism that decides at decode time which compressed spans to restore to token resolution. SeKV segments the context at high-surprisal positions, keeps anchor tokens and span summaries on GPU, and offloads low-rank factors of each span's keys and values to CPU. Lightweight per-head routers select query-relevant spans, which are reconstructed with a learned per-span rank. The cache is lossy but non-evicting: every span stays in attention, coarse or reconstructed. The base LLM stays frozen, and the router adds under 0.1% parameters. Across five backbones and four benchmarks at a 10% GPU-resident KV budget, SeKV outperforms every method that fixes resolution at prefill, improving over the strongest semantic baseline by 2.9 points (5.5% relative). It matches exact-recall offloading caches on LongBench and InfiniteBench and trails on synthetic exact retrieval, while transferring 3 times less data per decoding step than ShadowKV and using the least total memory among the methods we profile. At 128K context, it halves peak GPU memory and cuts total memory by one third relative to full KV caching.

Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2606.31145 [cs.CL]
	(or arXiv:2606.31145v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2606.31145 arXiv-issued DOI via DataCite

Submission history

From: Amirhossein Abaskohi [view email]
[v1] Tue, 30 Jun 2026 05:18:02 UTC (623 KB)
[v2] Wed, 4 Nov 2026 22:08:49 UTC (281 KB)