arXiv cs.AI· Ilya Lasy, Nora Yinuo Cai, Kola Ayonrinde·· 9 天前AI 评分25
RouterInterp：理解 MoE 路由中的叠加特化现象
RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
AI 导读
研究提出叠加特化假设（SSH），认为 MoE 专家并非专精单一领域，而是专精于细粒度特征的并集。基于该假设的方法 RouterInterp 在 gpt-oss-20b 上解释专家路由，检测准确率比此前基于 token 统计的方法高约 65%。该工作为生成更准确的专家路由解释提供了可扩展方法，论文已被 ICML 2026 收录。
正文 · 原文
Abstract:Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens. A leading hypothesis for the performance of MoE models is that each expert specialises in a single, coherent domain. However, interpretability efforts that assume this hypothesis have generally been unsuccessful. We propose and present evidence for an alternative account that we call the Superposed Specialisation Hypothesis (SSH): experts specialise in a disjoint union of fine-grained features rather than one broad domain. Leveraging the SSH, we introduce RouterInterp, a method for interpreting expert routing that identifies Sparse Autoencoder features most predictive of routing decisions and produces unified natural language explanations. On gpt-oss-20b, RouterInterp explains expert routing with ${\sim}65\%$ higher detection accuracy than prior token statistics based methods. This work provides a scalable method for generating more accurate explanations of expert routing and increases our understanding of a previously uninterpretable component of foundation models.
| Comments: | 33 pages (12 non-appendix pages), 7 figures, published as a conference paper at ICML 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.11775 [cs.AI] |
| (or arXiv:2610.11775v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11775 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ilya Lasy [view email]
[v1]
Thu, 8 Oct 2026 11:55:25 UTC (990 KB)
来源：arXiv cs.AI · arxiv.org