arXiv cs.AI· Bingcong Li, Hao Ma, Michael Muehlebach, Niao He, Yi Ma·· 2 天前AI 评分22
重新思考现代 MoE 中的路由探索：MirrorMoE 如何化解探索困境
Rethinking Routing Exploration in Modern Mixture-of-Experts
AI 导读
研究指出现代 MoE 架构中细粒度专家分割与共享专家之间存在"探索困境"：共享专家降低了 token 表示的多样性，削弱路由探索，而细粒度分割又因组合爆炸式扩大的路由空间更需要探索，导致其表达能力难以充分释放。为此提出的 MirrorMoE 通过 token 自适应抑制冗余专家来缩小探索空间，在不同模型规模上均稳定提升性能。
正文 · 原文
Abstract:Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.22353 [cs.LG] |
| (or arXiv:2610.22353v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22353 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bingcong Li [view email]
[v1]
Thu, 15 Oct 2026 17:58:02 UTC (779 KB)
来源：arXiv cs.AI · arxiv.org