arXiv cs.CL· Yifan Yang, Hanling Yi, Dongquan Yang, Feng Lin, Rong Xiao, Aijun Li, Xiaotian Yu·· 13 小时前AI 评分34
SPARE：用路由熵分配稀疏度的高效 MoE 专家剪枝方法
SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
AI 导读
SPARE 是一种无需训练、与选择器无关的 MoE 专家剪枝方法，通过校准数据上 token 归一化 Top-K 门控权重的路由熵，决定每层剪掉多少专家，路由越分散的层剪得越激进。
正文 · 原文
Abstract:Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment. Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally. We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data. When a token's gate weight is spread evenly, removing one active expert discards little of it, so SPARE prunes layers with more diffuse routing more aggressively. A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.29662 [cs.LG] |
| (or arXiv:2610.29662v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29662 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yifan Yang [view email]
[v1]
Wed, 21 Oct 2026 12:40:00 UTC (6,337 KB)
来源：arXiv cs.CL · arxiv.org