arXiv cs.CL· Ke Wan, Zaiyi Zheng, Chen Chen·· 12 小时前AI 评分30
SPARE：用紧凑循环状态实现快速线性注意力解码
SPARE: Fast Linear Attention Decoding with Compact Recurrent States
AI 导读
SPARE 是一种无需训练的方法，通过固定低维子空间、异构秩分配和轻量写入补偿，直接对紧凑循环状态进行读取与更新，避免在线分解和稠密状态重建。它实现 2.55 倍全模型解码加速、循环核 DRAM 流量降低 7.49 倍、峰值 GPU 内存降低 3.31 倍，并在同等内存预算下保持优于结构化剪枝的关联检索准确率。
正文 · 原文
Abstract:Linear attention reduces the quadratic sequence-length complexity of softmax attention by maintaining fixed-size recurrent states. However, these states scale quadratically with the representation dimension and require repeated data transfers during decoding, creating a substantial GPU memory bottleneck. Recent studies have revealed significant low-rank redundancy in recurrent states, suggesting opportunities for more efficient representations. We introduce SPARE (State Projection for Accelerated Recurrent Execution), a training-free method that exploits this redundancy through fixed low-dimensional subspaces, heterogeneous rank allocation, and lightweight write compensation. By performing recurrent reads and updates directly on compact states, SPARE reduces memory traffic without online decomposition or dense-state reconstruction. SPARE achieves 2.55x full-model decoding speedup, 7.49x lower recurrent-kernel DRAM traffic, and 3.31x lower peak GPU memory, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets.
| Comments: | 10 pages, 4 figures, 2 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.07552 [cs.CL] |
| (or arXiv:2611.07552v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07552 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ke Wan [view email]
[v1]
Thu, 5 Nov 2026 07:00:23 UTC (173 KB)
来源：arXiv cs.CL · arxiv.org