arXiv cs.AI· Ning Yang, Chengzhi Wang, Yibo Liu, Baoliang Tian, Haijun Zhang·· 11 小时前AI 评分35
CompilerKV：通过离线经验编译实现风险自适应 KV 压缩
CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation
AI 导读
CompilerKV 将 KV 保留策略的校正表从校准语料离线编译，把在线校正简化为 O(1) 查表加预算钳制。在 512-token 预算下，它在四个骨干模型上均达到压缩 SOTA，平均比最强 prefill-only 基线高 1.67 分；32k NIAH 上得分 0.89，SnapKV 为 0.42。
正文 · 原文
Abstract:Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction. The retention decision is therefore irreversible, yet existing methods estimate the corrective signals it relies on, per-head reliability and prompt-level compression sensitivity, online from a single noisy prompt. We argue this is the wrong statistical unit: these signals exhibit far higher cross-prompt regularity than within-prompt signal-to-noise. We introduce \textsc{CompilerKV}, a KV-retention policy whose corrective tables are compiled offline from a calibration corpus, reducing online correction after the standard observation-window scan to $O(1)$ lookups plus a budget clamp. We find that compiled retention tables behave as portable architectural priors: rankings transfer across disjoint corpora on four backbones (mean Spearman $\bar\rho{=}0.90$), and direct model-to-model table transfer costs only $0.4$--$0.8$ LongBench points on average. At a 512-token budget, \textsc{CompilerKV} attains compressed-SOTA on all four backbones, improving over the strongest prefill-only baseline by $+1.67$ points on average (task-bootstrap 95\% CI $[+1.08,+2.37]$). Pressure regimes amplify the gap: under a fixed $512/32k$ cache ratio, \textsc{CompilerKV} remains the strongest compressed method through 128k RULER ($\sim\!73$ vs.\ FullKV $\sim\!79$, SnapKV $\sim\!38$); on 32k NIAH it reaches $0.89$ vs.\ SnapKV $0.42$; and at 32k input, retaining only $1.56\%$ of the prefill KV, batch-16 serving remains feasible where FullKV is OOM.
| Comments: | Accepted by NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.08686 [cs.LG] |
| (or arXiv:2602.08686v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.08686 arXiv-issued DOI via DataCite |
Submission history
From: Yibo Liu [view email]
[v1]
Mon, 9 Feb 2026 14:07:55 UTC (2,492 KB)
[v2]
Tue, 19 May 2026 19:14:03 UTC (2,469 KB)
[v3]
Tue, 21 Jul 2026 10:42:28 UTC (2,482 KB)
[v4]
Wed, 4 Nov 2026 12:45:47 UTC (2,484 KB)
来源：arXiv cs.AI · arxiv.org