arXiv cs.AI· Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang·· 14 小时前AI 评分52
研究系统对比十种预训练优化器：Muon、Soap 加速比随模型规模收窄
Fantastic Pretraining Optimizers and Where to Find Them
AI 导读
论文对十种深度学习优化器在 0.1B-1.2B 参数、1-8 倍 Chinchilla 最优数据模型比下做了系统对比，指出超参数调优不平等和评测设置有限是此前比较失真的两个原因。
正文 · 原文
Abstract:AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups. To address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training. First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. Second, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1x for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay. Through our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners -- multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.
| Comments: | 108 pages, 8 figures, reproducible runs available at this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2509.02046 [cs.LG] |
| (or arXiv:2509.02046v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.02046 arXiv-issued DOI via DataCite |
Submission history
From: Kaiyue Wen [view email]
[v1]
Tue, 2 Sep 2025 07:43:22 UTC (701 KB)
[v2]
Thu, 4 Sep 2025 19:22:04 UTC (707 KB)
[v3]
Sun, 25 Oct 2026 04:22:04 UTC (707 KB)
来源：arXiv cs.AI · arxiv.org