arXiv cs.CL· Zelin Zhao, Junyuan Shang, Tianchi Ma, Shuohuan Wang, Dianhai Yu·· 12 小时前AI 评分29
Multi-View Latent MoE：让每个路由器塑造专家计算
Multi-View Latent MoE: Letting Every Router Shape Expert Computation
AI 导读
研究者提出 Multi-View Latent MoE，用多个路由器聚合 sigmoid 分数改进专家选择，并复用路由偏好调制专家输入。在 24 层 10B-A1B 模型、708B token 训练下，九任务下游均值较基线提升 1.18 个百分点，仅 Multi-Router 即带来 0.6 个百分点增益。该方法保持激活专家数、专家 FLOPs 与 all-to-all 通信量不变。
正文 · 原文
Abstract:Sparse mixture-of-experts (MoE) models expand parameter capacity by activating a small subset of experts per token. Expert selection directly affects model capabilities under a fixed active-expert budget. Dense evaluation of all experts reveals weak agreement between single-router rankings and expert-utility evaluation rankings, especially in middle and deep layers. Richer signals for selection and computation incur additional cost: selection based on internal expert activations requires preliminary computation, while processing multiple full representations separately through the same expert repeats its computation. We propose Multi-View Latent MoE to improve expert selection with multiple routers and reuse their preferences to modulate expert computation under a fixed expert budget. Its Multi-Router submodule aggregates sigmoid scores from independently initialized routers. These routers develop highly differentiated expert preferences during training, allowing experts strongly favored by one router or jointly supported by several routers to be selected. Multi-View further uses each selected expert's relative scores across routers to mix learned latent views into its input, reusing routing preferences for both selection and computation. Each selected expert executes one forward pass, preserving the active-expert count, expert FLOPs, and all-to-all communication volume. On a 24-layer 10B-A1B model trained on 708B tokens, Multi-View Latent MoE improves the nine-task downstream mean by 1.18 percentage points over the baseline. Multi-Router alone yields a 0.6-percentage-point gain and improves the rank correlation between router scores and evaluated expert utility, with relative increases of 8.1% globally and approximately 58% in the layer with the lowest single-router alignment. Controlled mixing interventions show the model uses input differences that depend on the expert and token.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.40577 [cs.CL] |
| (or arXiv:2610.40577v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40577 arXiv-issued DOI via DataCite |
Submission history
From: Zelin Zhao [view email]
[v1]
Thu, 29 Oct 2026 17:45:12 UTC (558 KB)
[v2]
Mon, 2 Nov 2026 17:49:38 UTC (558 KB)
来源：arXiv cs.CL · arxiv.org