arXiv cs.CL· Utkarsh Kanwat·· 13 小时前AI 评分34
将注意力偏置折叠进 Key：压缩 KV 缓存可直接跑在标准内核上
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels
AI 导读
研究者提出把 Attention Matching 的逐 key 偏置沿每个 head 的一条闭式方向折叠进 key，使压缩后的 KV 缓存变成普通缓存，从而能被 FlashAttention 和 vLLM 的 paged cache 直接使用。
正文 · 原文
Abstract:Attention Matching (AM) compacts a KV cache by keeping some keys per head, refitting their values and adding a bias to each kept key's logit. FlashAttention and vLLM's paged caches have no input for it. We fold the bias into the keys along one closed-form direction per head, fitted on context queries, giving a plain cache with AM's size and head dimension and the model's own query path; the output error is bounded by how far each query's projection on that direction is from 1. With equal per-head budgets, unmodified vLLM serves folded caches through a KV connector, matching Hugging Face's answers on all 51 Qwen3-4B questions and its greedy output on 180 of 200 Qwen3-14B LongBench samples. With equal budgets on QuALITY (Qwen3-1.7B, 4B, 8B, Llama-3.1-8B) and LongHealth (three Qwen3 models), the fold is within noise of AM's accuracy in 13 of 14 settings, with lower KL to the full cache than dropping the bias in all 12 where both ran. Under the published per-head budgets, on Qwen3-4B, its Instruct-2507 variant and Llama-3.1-8B, it is within noise of AM in 9 of 10, no longer beats dropping the bias in KL, and the two each win once in accuracy.
| Comments: | 30 pages, 2 figures, 18 tables |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.36520 [cs.LG] |
| (or arXiv:2610.36520v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36520 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Utkarsh Kanwat [view email]
[v1]
Tue, 27 Oct 2026 02:22:37 UTC (87 KB)
来源：arXiv cs.CL · arxiv.org