arXiv cs.AI· Zirui Peng, Yizhou Liu, Ziming Liu, Jeff Gore·· 9 天前AI 评分26
注意力机制中的非线性如何导致逆深度缩放
Emergent Inverse-Depth Scaling From Nonlinearity In Attention
AI 导读
研究发现，非线性注意力会导致损失随深度呈逆深度衰减，在所有测试的数据谱上均成立。非线性使注意力能选择性聚焦相关 token，让强、弱谱方向并行学习，这与线性注意力模型依赖全局谱强度、按强弱顺序学习的机制不同。作者据此提出，深度缩放可能源于注意力的非线性，使大语言模型能局部聚焦，从而让全局协方差结构变得不那么重要。
正文 · 原文
Abstract:Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.
| Comments: | 30 pages, 14 figures, 5 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11063 [cs.LG] |
| (or arXiv:2610.11063v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11063 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zirui Peng [view email]
[v1]
Thu, 8 Oct 2026 01:23:54 UTC (2,292 KB)
来源：arXiv cs.AI · arxiv.org