arXiv cs.CL· Fang Wan, Xufeng Liu, Fan Li, Yi Liu·· 9 天前AI 评分22
稀疏注意力是矩阵近似，而非从一堆数值中挑选
Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values
AI 导读
针对稀疏注意力仅保留注意力矩阵中大数值或高权重区域的错误做法，研究者提出 Matrix Approximation Sparse Attention（MASA），用闭式评分衡量每个稀疏单元对矩阵乘积近似误差的削减量。MASA 可作为即插即用修正项加入现有稀疏注意力框架，无需改动稀疏 kernel 或预算，在多种稀疏注意力方法、benchmark 和 LLM 主干上取得一致的准确率提升。
正文 · 原文
Abstract:Large Language Models (LLMs) achieve strong performance across many domains, but their efficiency is limited by the quadratic cost of attention with respect to prompt length. Sparse attention reduces this cost by retaining only a small fraction of query-key interactions to approximate the full attention matrix. However, existing methods are trapped in a mathematically wrong view: they simply keep large scalar entries or high-mass regions of the attention matrix. This treats the attention matrix as a bag of values, ignoring that it is used as a structured matrix whose entries jointly determine the attention output through multiplication with value vectors. We argue that this is the core conceptual issue: sparse attention should be formulated as matrix approximation, not as blindly choosing the largest values from a bag of entries. Based on this view, we propose Matrix Approximation Sparse Attention (MASA). MASA replaces raw attention-mass ranking with a closed-form score that measures how much each sparse unit reduces matrix-product approximation error. As a theory-grounded plug-in correction, MASA can be added to existing sparse attention frameworks without changing their sparse kernels or budgets. Extensive experiments across multiple sparse attention methods, benchmarks, and LLM backbones show consistent accuracy gains, supporting both MASA and the matrix-approximation view of sparse attention.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.10871 [cs.CL] |
| (or arXiv:2610.10871v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10871 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fang Wan [view email]
[v1]
Wed, 7 Oct 2026 20:17:13 UTC (226 KB)
来源：arXiv cs.CL · arxiv.org