arXiv cs.AI· Jihwan Kim, Dogyoon Song, Chulhee Yun·· 9 天前AI 评分29
从几何到泛化：行归一化为何能胜过 Adam 和 Muon
From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon
AI 导读
研究表明，在高维多分类任务中，行归一化（row-wise normalization）的总体准确率严格高于全批量 Adam 和精确 SVD Muon。其优势源于行归一化的类间欧氏几何能渐近保留总体决策边界方向，而 Adam 的坐标几何与 Muon 的谱几何会引入不可忽略的扭曲。该优势在幂律谱、强各向异性噪声下依然成立，合成实验与语言模型末层实验均予支持。
正文 · 原文
Abstract:Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear. We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification. Under an isotropic Gaussian-cloud data model, this advantage arises because row normalization's class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions, whereas Adam's coordinate-wise geometry and Muon's spectral geometry introduce nonvanishing distortions. Beyond isotropy, the advantage persists for full-batch training on class means with independently oriented class-mean and test-noise covariances. It holds for power-law spectra with class-mean exponent below one, even under heavily anisotropic test noise. When both covariances are diagonal and sufficiently close, the advantage over Adam can reverse, while applying the same random rotation to both restores it by changing only their alignment with Adam's coordinate axes. Synthetic and last-layer language-model experiments support the predicted advantage.
| Comments: | 90 pages, 9 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.11309 [cs.LG] |
| (or arXiv:2610.11309v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11309 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jihwan Kim [view email]
[v1]
Thu, 8 Oct 2026 06:15:13 UTC (587 KB)
来源：arXiv cs.AI · arxiv.org