arXiv cs.AI· Yaoxiang Wang, Kai Zhang, Ruobing Xie, Lanrui Wang, Guoliang Zhao, Junhui Li, Jinsong Su·· 3 天前AI 评分24
LatentMoE：从专家几何到逐层容量分配的专家形状设计
Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
AI 导读
LatentMoE 将专家计算移入低维空间，使专家的输入维度与中间宽度成为显式设计选项，并证明专家局部 Jacobian 秩受这两个维度共同约束，在固定单专家参数量下二者均衡可最大化秩上限。跨模型规模和路由粒度的形状扫描一致偏好近正方形专家；进一步用逐层内在维度（ID）指导深度方向的容量分配，在保持每层参数量、激活 FLOPs 和通信量不变的情况下，相比逐层同构基线取得更低验证损失和更强下游性能。
正文 · 原文
Abstract:Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing granularity. LatentMoE moves expert computation into a lower-dimensional space, exposing the input dimension and intermediate width of each expert as explicit design choices. We show that an expert's local Jacobian rank is jointly bounded by these two dimensions, and that balancing them maximizes the rank ceiling at fixed per-expert parameters. Controlled shape sweeps across model scales and routing granularities consistently favor near-square experts. We further use layerwise intrinsic dimension (ID) to guide capacity allocation across depth, co-varying the number and shape of experts while keeping per-layer parameter count, activated FLOPs, and communication volume constant. Compared with layer-homogeneous baselines, our ID-guided allocation achieves lower validation loss and stronger downstream performance. These results establish expert shape and layerwise allocation as practical design axes for MoEs.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.14644 [cs.AI] |
| (or arXiv:2610.14644v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14644 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaoxiang Wang [view email]
[v1]
Sat, 10 Oct 2026 06:54:36 UTC (199 KB)
来源：arXiv cs.AI · arxiv.org