arXiv cs.CL· Masato Fujitake·· 13 小时前AI 评分31
EnergyCal：抑制 Muon 系列优化器在稀疏 MoE 训练中的损失尖峰
EnergyCal: Taming Loss Spikes in Muon-Family Optimizers for Sparse MoE Training
AI 导读
EnergyCal 是一种轻量规则，按路由专家有限步 Newton-Schulz 方向的归一化平方 Frobenius 范数衰减其更新，不改变谱变换、不增加持久优化器状态。
正文 · 原文
Abstract:Sparse mixture-of-experts (MoE) language-model training can exhibit large loss spikes with Muon-family optimizers. We introduce EnergyCal, a lightweight rule that attenuates routed-expert updates according to the normalized squared Frobenius norm of their finite-step Newton-Schulz directions. EnergyCal leaves the spectral transform unchanged and adds no persistent optimizer state. We evaluate it on a 4B-parameter MoE model. Against a development-selected, lower-learning-rate MuonH recipe, EnergyCal-MuonH reduces mean thresholded loss-spike burden by 99.51% and final geometric-mean perplexity by 4.06%, with a modest throughput cost. With plain Muon at matched nominal learning rate and weight decay, the reduction in mean spike burden averages 95.98% across the two tested exponents, accompanied by small endpoint improvements. These findings support stabilization both with and without Hyperball in the evaluated setting.
| Comments: | 9 pages with appendices |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.37062 [cs.LG] |
| (or arXiv:2610.37062v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37062 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Masato Fujitake [view email]
[v1]
Tue, 27 Oct 2026 11:31:17 UTC (481 KB)
来源：arXiv cs.CL · arxiv.org