arXiv cs.AI· Yibang Li, Bihari Lal Pandey, Ravi Sah, Andi Han, Cyrus Mostajeran, Pratik Jawanpuria, Bamdev Mishra·· 4 小时前AI 评分26
iMuon:黎曼矩阵流形上的内在谱优化
iMuon: Intrinsic Spectral Optimization on Riemannian Matrix Manifolds
AI 导读
研究者提出 iMuon,将 Muon 优化器推广到黎曼矩阵流形,在固定秩、SPD、Stiefel 和 Grassmann 流形上给出谱范数、Frobenius 范数和核范数的闭式更新。
正文 · 原文
Abstract:Muon and related norm-constrained optimizers, now central to large-scale learning, compute updates via a linear maximization oracle (LMO) over a matrix-norm ball in Euclidean space. However, they do not generalize cleanly to manifold-valued parameters such as low-rank factorizations, orthogonality constraints, or symmetric positive-definite (SPD) matrices. Naively restricting the Muon LMO to the tangent space (i) breaks quotient symmetries and (ii) couples the tangent-space constraint with an ambient norm bound, which obstructs closed-form solutions. We resolve both issues with a key observation: every Riemannian metric canonically lifts a unitarily invariant Euclidean norm to an intrinsic norm on each tangent space. The resulting intrinsic LMO is symmetry preserving and, under a structural condition on the scaled tangent space, reduces to a standard Euclidean LMO. Building on this, we introduce intrinsic Muon (iMuon), a unified framework with closed-form updates on the fixed-rank, SPD, Stiefel, and Grassmann manifolds for various unitarily invariant norms, including the spectral, Frobenius, and nuclear norms. We establish deterministic and stochastic convergence rates of iMuon with constants that depend only on the manifold dimension. Notably, on the fixed-rank manifold, the constant depends only on the rank, which makes the rate independent of factor conditioning and removes the runtime factor rescaling required by prior work. Experiments on LoRA fine-tuning of LLMs, image classification, and subspace learning illustrate the efficacy of the proposed approach.
| Comments: | Code: this https URL Accepted at NeurIPS 2026. Updated to match the camera-ready version |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.09238 [cs.LG] |
| (or arXiv:2605.09238v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.09238 arXiv-issued DOI via DataCite |
Submission history
From: Yibang Li Nanyang technological university [view email]
[v1]
Sun, 10 May 2026 00:39:13 UTC (356 KB)
[v2]
Mon, 16 Nov 2026 23:41:40 UTC (272 KB)
来源:arXiv cs.AI · arxiv.org