arXiv cs.AI· Tianqi Chen, Xunchao Li, Linchuan Xie, Guanghua Yu, Jianchen Zhu, Peisong Wang, Jian Cheng·· 13 小时前AI 评分38
RAQ-MLA：面向多头潜在注意力缓存的角色感知量化
RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches
AI 导读
RAQ-MLA 提出一种无需训练的 MLA KV 缓存量化方法，可将完整缓存量化至约 2.5 bits/元素，理论缓存容量提升 6.5 倍。该方法针对 MLA 缓存的两个张量分别设计可逆变换：潜在向量主要承载 value，按注意力加权 value 误差拟合；位置 key 主导注意力分数，按分数扰动拟合。在四个 MLA 模型上，推理与编码任务几乎无损。
正文 · 原文
Abstract:Multi-Head Latent Attention (MLA) has been adopted by recent frontier large language models. Instead of per-head keys and values, MLA caches a compressed KV latent that is up-projected into keys and values at decode time, together with a narrow positional key that carries position information into the attention score. Existing KV-cache quantization methods are designed around the keys and values of standard attention and do not account for what MLA actually caches, which costs accuracy at low bit widths. We analyze the MLA cache both in activation distribution and in how quantization error propagates to the attention output, and show that the two cached tensors play different roles: the latent mainly supplies the values, whereas the positional key dominates the attention score. We therefore propose RAQ-MLA (Role-Aware Quantization for Multi-Head Latent Attention Caches), which fits an invertible transform to each cached tensor against the error that tensor delivers to the attention output: attention-weighted value error for the latent and score perturbation for the positional key. Without training, RAQ-MLA quantizes the full MLA cache to about 2.5 bits per element, a 6.5x increase in theoretical KV-cache capacity, while remaining nearly lossless on reasoning and coding tasks across four MLA models.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.33625 [cs.AI] |
| (or arXiv:2610.33625v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33625 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chen Tianqi [view email]
[v1]
Sat, 24 Oct 2026 11:54:55 UTC (7,327 KB)
来源：arXiv cs.AI · arxiv.org