arXiv cs.CL· Esther Xin·· 11 小时前AI 评分31
先调好基线：Muon 缩放权重衰减的大部分收益来自调参差距，但其小幅真实优势随模型规模增长
Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size
AI 导读
研究在 MAI-Base-2 协议下重新对比 Muon-SW 与恒定解耦权重衰减，发现共享超参时 Muon-SW 领先 0.035 BPB（1.52x 算力节省），双方各自调参后差距仅 0.0026 BPB（1.03x），约 90% 优势消失。
正文 · 原文
Abstract:Scaled weight decay for Muon (Muon-SW) has been reported to train language models more efficiently than constant decoupled weight decay, but such comparisons usually reuse one learning-rate (LR) and weight-decay (WD) setting for both methods. We re-run the comparison under the MAI-Base-2 protocol: LR and WD are tuned jointly through their product for each method, on a source-scaled mixture-of-experts (MoE) scaling ladder (1.9M to 15M active parameters) that holds tokens per active parameter and data repetition fixed across sizes. We find: (1) Most of the gain is a tuning gap. At the shared setting Muon-SW wins by 0.035 bits per byte (BPB), a 1.52x compute saving; after both methods are tuned, the gap is 0.0026 BPB (95% CI [0.0018, 0.0034]), a 1.03x saving, so about 90% of the advantage disappears. (2) The small advantage that remains is real: at the smallest size, all four paired seeds favour Muon-SW at the tuned settings. (3) That advantage grows with model size: across an 8x range of active parameters, the tuned gap rises from 0.0036 to 0.0051 BPB while the untuned gap shrinks from 0.035 to 0.024, so the surviving share rises from about 10% to 21%. (4) The best weight decay moves with scale: the optimal LRWD product grows for both methods, faster for Muon-SW (ratio 1.40 to 1.91), and a law fitted on the two smallest sizes overshoots the third by 1.3-1.5x, costing up to 0.002 BPB. The whole study runs on one NVIDIA T4 GPU; we release the code and all run results.
| Comments: | 9 pages,8 figures, code and data:this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| ACM classes: | I.2.6; I.2.7 |
| Cite as: | arXiv:2611.02529 [cs.CL] |
| (or arXiv:2611.02529v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02529 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Esther Xin [view email]
[v1]
Mon, 2 Nov 2026 04:59:08 UTC (311 KB)
来源：arXiv cs.CL · arxiv.org