arXiv cs.CL· Hanqing Liu, Jianjun Cao, Yuanze Li, Zijian Zhou·· 9 天前AI 评分22
Grokking 还是 Glitching？低精度如何驱动 Slingshot 损失尖峰
Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
AI 导读
论文证明深度网络长期无正则训练中的 Slingshot 损失尖峰源于浮点运算精度极限：训练进入高置信阶段后，正确类 logit 与其他 logit 之差超过吸收误差阈值，反向传播中正确类梯度被舍入为零，破坏各类梯度的零和约束，引发分类器层参数系统性漂移。
正文 · 原文
Abstract:Deep neural networks exhibit periodic loss spikes during unregularized long-term training, a phenomenon known as the "Slingshot Mechanism." Existing work usually attributes this to intrinsic optimization dynamics, but its triggering mechanism remains unclear. This paper proves that this phenomenon is a result of floating-point arithmetic precision limits. As training enters a high-confidence stage, the difference between the correct-class logit and the other logits may exceed the absorption-error threshold. Then during backpropagation, the gradient of the correct class is rounded exactly to zero, while the gradients of the incorrect classes remain nonzero. This breaks the zero-sum constraint of gradients across classes and introduces a systematic drift in the parameter update of the classifier layer. We prove that this drift forms a positive feedback loop with the feature, causing the global classifier mean and the global feature mean to grow exponentially. We call this mechanism Numerical Feature Inflation (NFI). This mechanism explains the rapid norm growth before a Slingshot spike, the subsequent reappearance of gradients, and the resulting loss spike. We further show that NFI is not equivalent to an observed loss spike: in more practical tasks, partial absorption may not produce visible spikes, but it can still break the zero-sum constraint and drive rapid growth of parameter norms. Our results reinterpret Slingshot as a numerical dynamic of finite-precision training, and provide a testable explanation for abnormal parameter growth and logit divergence in late-stage training.
| Comments: | 29 pages, 13 figures; accepted to NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Optimization and Control (math.OC); Machine Learning (stat.ML) |
| Cite as: | arXiv:2605.06152 [cs.LG] |
| (or arXiv:2605.06152v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06152 arXiv-issued DOI via DataCite |
Submission history
From: Hanqing Liu [view email]
[v1]
Thu, 7 May 2026 12:45:21 UTC (893 KB)
[v2]
Tue, 12 May 2026 10:56:27 UTC (893 KB)
[v3]
Tue, 26 May 2026 10:55:47 UTC (893 KB)
[v4]
Wed, 7 Oct 2026 23:29:06 UTC (893 KB)
来源：arXiv cs.CL · arxiv.org