跳到正文
arXiv cs.CL· Md Rifat Arefin, Ravid Shwartz-Ziv, Ernie Chang, Chinnadhurai Sankar, Rylan Conway, Aristide Baratin, Irina Rish, Adithya Sagar, Patrick Huber·· 2 天前AI 评分31

Muon 预训练中临时权重衰减的延迟效应研究

Delayed Effects of Temporary Weight Decay in Muon Pretraining

AI 导读

在 Muon 训练的 162M 至 3.4B 参数 decoder-only Transformer 上,临时权重衰减窗口结束后验证损失可能先升后降。162M 选定的调度迁移到 1.4B 后,移除时比无衰减差 0.26 nats,训练进行到 73% 才反超,最终好 0.028 nats。

正文 · 原文

View PDF HTML (experimental)

Abstract:A temporary optimization intervention can increase validation loss when it ends yet reduce it at the final training budget. We demonstrate this with temporary weight decay in Muon-trained decoder-only Transformers with 162M, 405M, 1.4B, and 3.4B parameters. For models up to 1.4B parameters, moving a fixed-duration decay window from early to late training reverses its final effect. At 162M, changing window duration alters final loss even at approximately matched total shrinkage. Most strikingly, a schedule selected at 162M and transferred without retuning to 1.4B is 0.26 nats worse than no decay when removed, overtakes it only after 73% of training, and ultimately finishes 0.028 nats better while also outperforming the best sampled constant-decay baseline. Early decay leaves a persistent weight-norm deficit, which keeps relative steps larger after the intervention ends. An approximate norm-dynamics recurrence captures the slow convergence of these relative steps, and fixed-norm replays of the measured trajectories reproduce the performance ordering and a similar loss difference. At 162M and 405M, norm-reset experiments show that smaller weight norms at the end of the decay window explain most of the final loss improvement. A separate Gram intervention exhibits related post-removal norm and relative-step dynamics. These results show that temporary optimization schedules can have delayed effects long after removal and should therefore be evaluated at the intended final training budget, not only at removal.
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as: arXiv:2611.17876 [cs.LG]
  (or arXiv:2611.17876v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.17876

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Md Rifat Arefin [view email]
[v1] Fri, 13 Nov 2026 08:05:19 UTC (450 KB)

来源:arXiv cs.CL · arxiv.org