arXiv cs.AI· Zaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum·· 14 小时前AI 评分38
量化误差如何引发语言模型预训练中的“定时炸弹”式损失尖峰
The Role of Quantization in Loss Spikes
AI 导读
语言模型预训练中会出现仅在长时间训练后才爆发的延迟性损失尖峰，研究称之为“定时炸弹尖峰”，其前兆是 LM head 量化误差快速增大。降低数值精度会加速这类尖峰，提高精度则改善稳定性；将 LM head 上的 AdamW 换成带动量的 SGD 可抑制 logit 增长、提升长程训练稳定性。现有各类缓解方法均能降低量化误差，指向共同的数值机制。
正文 · 原文
Abstract:Loss spikes in language model pretraining can waste substantial compute and derail otherwise healthy training runs. A primary driver of these instabilities is the growth and divergence of output logits in the final language modeling (LM) head. In this work, we study a particularly costly form of this instability associated with output logit growth: delayed, terminal divergences that emerge only after prolonged training, which we call time-bomb spikes. We find that these instabilities are preceded by rapidly growing quantization error in the LM head, which eventually culminates in a loss spike. Crucially, reducing numerical precision accelerates time-bomb spikes, while increasing precision improves stability. We then propose a simple yet effective mitigation that replaces AdamW on the LM head with SGD with momentum, suppressing logit growth and improving stability over long training horizons. Finally, we find that existing mitigation techniques from the literature (along with our own) all reduce quantization error, suggesting a shared numerical mechanism behind their effectiveness. Together, our results identify finite precision error as a hidden driver of time-bomb spikes and show that suppressing this error is key to stable long-horizon pretraining.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38277 [cs.LG] |
| (or arXiv:2610.38277v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38277 arXiv-issued DOI via DataCite |
Submission history
From: Zaiqian Chen [view email]
[v1]
Wed, 28 Oct 2026 04:53:13 UTC (2,078 KB)
来源：arXiv cs.AI · arxiv.org