arXiv cs.CL· Wenwen Hou, Xinyuan Song, Wenqian Weng, Shiwei Liu·· 12 小时前AI 评分26
ActTail：基于全局激活稀疏性的大语言模型推理加速方法
ActTail: Global Activation Sparsity in Large Language Models
AI 导读
ActTail 提出一种基于 Heavy-Tailed Self-Regularization（HT-SR）理论的全局激活稀疏分配方法，按各投影层重尾指数分配差异化稀疏预算，替代统一稀疏策略。在 LLaMA 和 Mistral 上，80% 稀疏度下 LLaMA-2-7B 困惑度降低 21.8%、LLaMA-2-13B 降低 40.1%、Mistral-7B 降低 9.4%，并同步提升下游任务表现。
正文 · 原文
Abstract:Activation sparsity is a promising approach for accelerating large language model (LLM) inference by reducing computation and memory movement. However, existing activation sparsity methods typically apply uniform sparsity across projections, ignoring the heterogeneous statistical properties of Transformer weights and thereby amplifying performance degradation. In this paper, we propose ActTail, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory. Specifically, we capture this heterogeneity via the heavy-tail exponent computed from each projection's empirical spectral density (ESD), which is used as a quantitative indicator to assign projection-specific sparsity budgets. Importantly, we provide a theoretical analysis that establishes an explicit relationship between the activation sparsity ratio and the heavy-tail exponent under the HT-SR regime, offering principled guidance for sparsity allocation beyond heuristic design. Experiments on LLaMA and Mistral models show that our method improves both perplexity and downstream task performance at high sparsity compared to uniform allocation. At 80% sparsity, perplexity is reduced by 21.8% on LLaMA-2-7B, 40.1% on LLaMA-2-13B, and 9.4% on Mistral-7B.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2603.12272 [cs.CL] |
| (or arXiv:2603.12272v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.12272 arXiv-issued DOI via DataCite |
Submission history
From: Wenwen Hou [view email]
[v1]
Wed, 18 Feb 2026 14:46:03 UTC (336 KB)
[v2]
Mon, 26 Oct 2026 11:50:07 UTC (338 KB)
来源：arXiv cs.CL · arxiv.org