arXiv cs.AI· Minjia Mao, Shi Chen, Bowen Yin, Xiao Fang·· 10 天前AI 评分24
大语言模型中的大规模激活门控通道（MAGC）
Massive Activation Gating Channel in Large Language Models
AI 导读
研究发现，大语言模型中大规模激活（massive activations）的出现由 spike FFN 输入嵌入中的单一通道控制，该通道位置对特定 LLM 固定，被命名为大规模激活门控通道（MAGC）。
正文 · 原文
Abstract:Massive activations, a phenomenon in which a small number of hidden channels exhibit exceptionally large magnitudes, are pervasive in large language models (LLMs). However, the mechanism by which a token develops massive activations as it propagates through a pretrained LLM remains poorly understood. In this paper, we find that the emergence of massive activations is controlled by a single channel in the input embedding to a spike feed-forward network (FFN). The position of this channel is fixed for a particular LLM. We name this channel the massive activation gating channel (MAGC). When the value of the MAGC is sufficiently large (or small, depending on the LLM), the output of the spike FFN exhibits massive activations. Examining six LLMs across four model families and different model sizes, we verify the existence and effect of MAGC. We further provide a theoretical explanation of the mechanism by which MAGC induces massive activations. When the value of MAGC is sufficiently large (or small), the output of a spike FFN asymptotically reduces to a quadratic form that mixes a few columns of the down-projection matrix of the FFN. Since these columns exhibit the shape of massive activations, the output therefore exhibits massive activations.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07661 [cs.AI] |
| (or arXiv:2610.07661v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07661 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Minjia Mao [view email]
[v1]
Tue, 6 Oct 2026 02:55:48 UTC (4,847 KB)
来源：arXiv cs.AI · arxiv.org