arXiv cs.AI· Tian Wang, Yongding Tao, Yihong Dong, Xiaoming Wu, Ying Guo, Ge Li·· 2 天前AI 评分33
专家存在结构化冗余：面向 MoE 大语言模型的结构保持专家剪枝
Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models
AI 导读
研究者发现 MoE 层专家输出存在结构化冗余，其稳定秩与熵有效秩远低于专家名义数量，据此将专家剪枝重构为结构保持的专家子集选择问题，并提出 PEEP 与 DCEP 两种剪枝方法。在 DeepSeek-V2-Lite 和 Qwen3-30B-A3B 上，75% 与 50% 专家保留率下，两方法在多数基准上优于 SOTA 基线，并在通用任务与数学推理任务上取得最佳平均性能。
正文 · 原文
Abstract:Sparse MoE mechanism significantly improves the trade-off between model capacity and inference efficiency. However, practical deployment still requires storing all experts, resulting in substantial memory overhead. We show that expert outputs in MoE layers exhibit structured redundancy. Across different models and task categories, the stable rank and entropy effective rank of expert outputs are consistently far below the nominal number of experts, while expert output distributions also exhibit fewer effective modes. Motivated by these observations, we reformulate expert pruning as a structure-preserving expert subset selection problem and efficiently optimize it via a greedy strategy, rather than only independently scoring and ranking individual experts or exhaustively searching over expert combinations. We propose two expert pruning methods under the unified structure-preserving framework: Projection Energy Expert Pruning (PEEP), which preserves the functional subspace by maximizing weighted projection energy, and Distributional Coverage Expert Pruning (DCEP), which preserves the distributional geometry by minimizing weighted coverage error. Both methods incorporate router-derived weights as priors, enabling them to jointly preserve the global structural relationships among experts while accounting for the local importance of individual experts. Experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B demonstrate that, at the expert retention ratios of 75% and 50%, our methods outperform the state-of-the-art baselines on the majority of benchmarks and consistently achieve the best average performance across both general tasks and mathematical reasoning tasks.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41236 [cs.AI] |
| (or arXiv:2610.41236v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41236 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tian Wang [view email]
[v1]
Fri, 30 Oct 2026 04:21:25 UTC (1,011 KB)
来源：arXiv cs.AI · arxiv.org