arXiv cs.CL· Jang-Hyun Kim, Dongyoon Han, Sangdoo Yun·· 14 小时前AI 评分38
Fast KVzip：用门控 KV 驱逐实现高效准确的 LLM 推理
Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
AI 导读
Fast KVzip 提出一种基于门控的 KV cache 驱逐方法，可在冻结权重 LLM 上驱逐最多 70% 的 KV cache，同时保持接近无损的性能。该方法引入轻量 sink-attention 门控模块识别并保留关键 KV 对，可无缝接入 prefill 与解码阶段，门控训练仅依赖前向传播、无需反向传播。
正文 · 原文
Abstract:Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.
| Comments: | NeurIPS 2026. Source code: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2601.17668 [cs.LG] |
| (or arXiv:2601.17668v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.17668 arXiv-issued DOI via DataCite |
Submission history
From: Jang-Hyun Kim [view email]
[v1]
Sun, 25 Jan 2026 03:07:54 UTC (139 KB)
[v2]
Mon, 9 Feb 2026 04:41:18 UTC (140 KB)
[v3]
Sun, 25 Oct 2026 16:41:39 UTC (198 KB)
来源：arXiv cs.CL · arxiv.org