arXiv cs.AI· Ermo Hua, Xiangyu Hong, Baiting Wu, Che Jiang, Haozheng Hou, Yicheng Gu, Hanjing Wang, Qirui Zeng, Ming Zhang, Ning Ding, Qipeng Guo, Youbang Sun, Biqing Qi, Bowen Zhou·· 12 小时前AI 评分26
Meta Expert：面向存储与检索高效 MoE 的跨层专家共享
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
AI 导读
研究提出跨层共享的 Meta Expert，替代主流分层架构中彼此独立但内容高度相似的 Local Expert。在激活专家数量与专家粒度固定时，该模型仅用 70–80% 的专家即可达到 Vanilla MoE 的性能，并表现出更低的参数冗余、更低的交叉熵损失和更高的下游任务准确率。理论分析显示 Meta Expert 还能为隐藏状态提供更灵活的残差路径，提升信息检索效率。
正文 · 原文
Authors:Ermo Hua, Xiangyu Hong, Baiting Wu, Che Jiang, Haozheng Hou, Yicheng Gu, Hanjing Wang, Qirui Zeng, Ming Zhang, Ning Ding, Qipeng Guo, Youbang Sun, Biqing Qi, Bowen Zhou
Abstract:As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged \textbf{Local Expert} in existing mainstream models, we hypothesize that inter-layer shared \textbf{Meta Expert} hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, \textbf{our model matches the performance of Vanilla MoE using only 70--80\% of the experts}, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.27598 [cs.AI] |
| (or arXiv:2610.27598v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27598 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ermo Hua [view email]
[v1]
Tue, 20 Oct 2026 04:05:56 UTC (254 KB)
来源：arXiv cs.AI · arxiv.org