arXiv cs.CL· Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho·· 10 小时前AI 评分40
将 MoE 剪枝并蒸馏为稠密语言模型的系统框架
Pruning and Distilling Mixture-of-Experts into Dense Language Models
AI 导读
研究者提出首个将训练好的 MoE 模型转换为标准全稠密架构的系统框架：对专家打分、选择并分组后拼接为稠密 FFN，再用知识蒸馏从 MoE 教师模型精炼。
正文 · 原文
Abstract:Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment. Existing compression methods reduce the number of experts but the output remains an MoE model with the same fundamental limitation. We present the first systematic framework for converting a trained MoE into a standard fully dense architecture: experts are scored, selected, and grouped, then concatenated into a dense FFN and refined by knowledge distillation from the MoE teacher. We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations. We find that the choice of scoring method is the most impactful, with our novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B. Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after ~4B-token distillation at 1.6x faster training wall-clock speed.
| Comments: | 44 pages, 8 figures, 28 tables. Accepted at NeurIPS 2026 (Oral) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.28207 [cs.CL] |
| (or arXiv:2605.28207v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.28207 arXiv-issued DOI via DataCite |
Submission history
From: Junhyuck Kim [view email]
[v1]
Wed, 27 May 2026 09:27:36 UTC (1,314 KB)
[v2]
Sat, 6 Jun 2026 13:23:43 UTC (1,396 KB)
[v3]
Tue, 10 Nov 2026 07:02:13 UTC (1,402 KB)
来源：arXiv cs.CL · arxiv.org