arXiv cs.CL· Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan·· 9 天前AI 评分27
ReadKV：面向 KV Cache 的查询自适应量化
Read What Matters: Query-Adaptive Quantization for KV Caches
AI 导读
ReadKV 提出一种查询自适应 KV cache 量化方法，将每个 key 和 value 存为渐进式编码，前缀支持不同重建精度，每个查询先按查询分配 key 通道前缀、再由重建 key 算出的注意力分配 value token 前缀，存储条目保持不变。
正文 · 原文
Abstract:KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. ReadKV stores each key and value in a progressive code whose prefixes support different reconstruction precisions. For each query, it allocates key-channel prefixes using the query, computes attention from the reconstructed keys, and then allocates value-token prefixes using that attention. Stored entries remain unchanged. Each stage optimizes a calibrated distortion objective under a fixed budget; we prove exact allocation under diminishing refinement gains and relate these objectives to attention-output error. We also exhibit a finite-dimensional attention family where query-dependent access strictly outperforms every query-independent reader at the same read budget, even with unrestricted competing encoders and decoders.
Across six base models, reading four bits on average from an eight-bit cache increases C4 perplexity by at most 0.66%, using about one quarter of the logical reads and half the retained capacity of a 16-bit cache. It is consistently more accurate than storing and fully reading four bits at the same payload-read budget. Retaining more bits than each query fetches is aimed at long-context decoding, where the cache bytes moved per step, rather than the weights, dominate cost. Long-context question answering and retrieval on two instruction-tuned models provide additional quality evidence. On the tested 8K-token, batch-one, single-layer workload on an NVIDIA A10G, a restricted eight-bit ReadKV reader with a two-bit mean payload-read budget has 39% lower latency than the tested TurboQuant codec.
| Comments: | 50 pages, 3 figures, 12 tables |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Performance (cs.PF) |
| Cite as: | arXiv:2610.11245 [cs.LG] |
| (or arXiv:2610.11245v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11245 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Siddharth Bhandari [view email]
[v1]
Thu, 8 Oct 2026 04:47:41 UTC (117 KB)
来源：arXiv cs.CL · arxiv.org