arXiv cs.CL· Hao Cheng (Zhejiang Gongshang University), Zhiyu Yang (Chang'an University), Bo Ma (Zhejiang Gongshang University), Hanzhang Jia (University of Sussex)·· 11 小时前AI 评分31
推理时专家停用需匹配训练期权重规则：MoE 模型停用一半专家后困惑度最高差 28740 倍
Matching the Training-Time Weight Rule in Inference-Time Expert Deactivation
AI 导读
研究对比移除、冻结、剩余专家重归一化和非归一化重路由四种推理时专家停用方式，在四个模型家族五个指令微调 checkpoint 及 OLMoE 基座上测试。按训练权重规则实施时困惑度为完整基线的 1.4 至 73 倍，规则不匹配时升至 14 至 28740 倍；OLMoE 上 GSM8K 严格匹配准确率在四种方式下均降至零。激活专家数越少，额外重归一化带来的困惑度惩罚越大。
正文 · 原文
Authors:Hao Cheng (Zhejiang Gongshang University), Zhiyu Yang (Chang'an University), Bo Ma (Zhejiang Gongshang University), Hanzhang Jia (University of Sussex)
Abstract:Expert deactivation at inference time changes both token routing and expert weights, and its effect on performance depends on how these changes are implemented. We compare removal, freezing, remaining-expert renormalization, and unnormalized re-routing on identical disabled sets across five instruction-tuned checkpoints from four model families, with a base OLMoE checkpoint as an additional comparison. We develop a weight-sum model that relates the loss of expert contributions and changes in branch scale to the weight rule used during training. With half of the experts deactivated according to activation-based rankings, remaining-expert renormalization yields the highest perplexity in models trained with raw router probabilities, whereas unnormalized re-routing yields the highest perplexity in models trained with normalized weights. Perplexity ranges from 1.4 to 73 times the intact baseline when the training rule is retained, and from 14 to 28,740 times the baseline under a rule mismatch. Task outcomes vary across evaluation types: on OLMoE, GSM8K strict-match accuracy falls to zero under all four implementations. Extending the model to changes in the number of active experts yields a further prediction, supported by the OLMoE experiments: the perplexity penalty from additional renormalization increases as fewer experts are activated.
| Comments: | Corresponding authors: Hanzhang Jia (lead corresponding) and Bo Ma |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.05804 [cs.CL] |
| (or arXiv:2611.05804v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05804 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanzhang Jia [view email]
[v1]
Wed, 4 Nov 2026 01:23:13 UTC (93 KB)
来源：arXiv cs.CL · arxiv.org