arXiv cs.CL· Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Jianchao Tan, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong·· 3 天前AI 评分29
OScaR：面向 LLM 及更多模型的极致 KV Cache 量化方案
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
AI 导读
针对 KV cache 显存瓶颈，研究者提出轻量压缩框架 OScaR（Omni-Scaled Canalized Rotation），指出 Token Norm Imbalance（TNI）是 per-channel 量化在极致压缩下失效的主因，并通过 Canalized Rotation 与 Omni-Token Scaling 缓解该问题。
正文 · 原文
Authors:Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Jianchao Tan, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong
Abstract:The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficient deployment. While the established per-channel quantization effectively accommodates intrinsic channel-wise outliers in Key tensors, its efficacy diminishes under extreme compression. In this work, we revisit the inherent limitations of the per-channel quantization paradigm from both empirical and theoretical perspectives. Our analysis identifies Token Norm Imbalance (TNI) as the primary bottleneck to quantization fidelity. We demonstrate that TNI systematically amplifies errors when shared quantization parameters are required to span token groups exhibiting substantial norm disparities. Instead of relying on intricate quantization pipelines (e.g., TurboQuant), we propose OScaR (Omni-Scaled Canalized Rotation), an accurate and lightweight KV cache compression framework for X-LLMs (i.e., text-only, multi-modal, and omni-modal LLMs). Advancing the per-channel paradigm, OScaR employs Canalized Rotation followed by Omni-Token Scaling to mitigate TNI-induced sequence-dimensional variance both effectively and efficiently, further supported by our optimized system design and CUDA kernels. Extensive evaluations across X-LLMs show that OScaR consistently outperforms existing methods and achieves near-lossless performance under INT2 quantization, establishing it as a robust, low-complexity, and universal framework that defines a new Pareto front. Compared with the BF16 FlashDecoding-v2 baseline, our OScaR implementation achieves a notable up to 3.0x speedup in decoding, reduces memory footprint by 5.3x, and increases throughput by 4.1x. The code for OScaR is publicly available at this https URL.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.19660 [cs.LG] |
| (or arXiv:2605.19660v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19660 arXiv-issued DOI via DataCite |
Submission history
From: Zunhai Su [view email]
[v1]
Tue, 19 May 2026 10:53:03 UTC (13,912 KB)
[v2]
Sat, 10 Oct 2026 04:19:44 UTC (13,937 KB)
[v3]
Wed, 14 Oct 2026 05:26:57 UTC (13,937 KB)
来源：arXiv cs.CL · arxiv.org