arXiv cs.CL· Yuhao Liu, Yuehao Li, Linghe Zhang, Gaobin Huang, Junran Li, Hongchang Wang, Haifang Jian·· 11 小时前AI 评分26
EarlySelectKV：用跨层查询代理提前完成 KV 选择
EarlySelectKV: Anticipatory KV Selection via Cross-Layer Query Proxies
AI 导读
EarlySelectKV 提出一种免训练方法，用较早隐藏状态的代理查询提前预计算稀疏 KV 选择索引，同时保留真实目标层查询用于最终注意力。在 RocketKV 主评测中，ES-RocketKV 在 LongBench 上相差 +0.018 分、RULER 约 -0.01 分、NIAH/Passkey 提升 0.25 个百分点。
正文 · 原文
Abstract:Query-conditioned sparse KV selection reduces long-context inference cost by attending to a subset of historical KV entries at each decoding step. Existing selectors must wait for the target-layer query before scoring candidates and generating indices, placing routing on the immediate pre-attention path. We introduce EarlySelectKV, a training-free transformation that uses proxy queries from earlier hidden states to precompute selected KV indices while retaining the real target-layer query for final attention. Tests with RocketKV, Quest, and Loki show no consistent pattern of quality degradation across the evaluated settings. In the primary RocketKV evaluation, ES-RocketKV differs by +0.018 points on LongBench, approximately -0.01 points on RULER, and +0.25 percentage points on representative NIAH/Passkey settings. In a TensorRT-LLM deployment with CPU-offloaded routing, ES-RocketKV improves end-to-end output throughput by up to 12.4% and reduces GPU-resident memory by 11.1% of the combined KV/K-summary pool. This makes routing timing an explicit optimization target alongside KV subset selection in sparse long-context decoding. Code is available at this https URL.
| Comments: | Published in EMNLP 2026. 20 pages. Code: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.01715 [cs.CL] |
| (or arXiv:2611.01715v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.01715 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pp. 5272-5291 (2026) |
| Related DOI: | https://doi.org/10.18653/2026.emnlp-main.229
DOI(s) linking to related resources |
Submission history
From: Yuhao Liu [view email]
[v1]
Sun, 1 Nov 2026 09:03:49 UTC (2,142 KB)
来源：arXiv cs.CL · arxiv.org