arXiv cs.AI· Haoyu Chen, Jun Bai, Yang Liu, Jianfei Zhang, Wenge Rong·· 11 小时前AI 评分32
MoEvolve：用同策略双向蒸馏强化 MoE 模型
A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation
AI 导读
研究者提出 MoEvolve 双向蒸馏框架，通过先恢复并强化激活扩展后的高-k MoE 模型、再将增强能力反向蒸馏回原始低-k 配置，在不增加推理激活预算的前提下提升模型性能。在多种 MoE 模型和基准上，该方法较初始模型最高提升 22.2 个领域平均分，并超过同等预算的 RLVR 基线最高 6.5 分，最终低-k 模型常反超其激活扩展教师模型。
正文 · 原文
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved Mixture-of-Experts (MoE) LLMs, but its outcome-level rewards provide only sparse supervision. On-Policy Distillation offers dense token-level feedback, yet requires a stronger compatible teacher, while on-policy self-distillation relies on hand-designed privileged information. We observe that the MoE architecture itself offers a way to resolve this dilemma, as increasing the activation budget per token exposes latent expert capacity and yields a potentially stronger teacher within the model. However, naive expansion degrades performance because newly co-activated experts are not optimized to work together. We therefore propose MoEvolve, a bidirectional distillation framework following a recovery-then-improvement strategy: we first restore the activation-expanded high-$k$ model and further strengthen it by exploiting its additional active capacity, then reverse the direction of distillation to transfer the enhanced capabilities back into the original low-$k$ configuration, retaining its inference-time activation budget. Across diverse MoE models and benchmarks, MoEvolve improves the initial models by up to $22.2$ domain-average points and outperforms same-budget RLVR baselines by up to $6.5$ points. Notably, the final low-$k$ models often surpass their activation-expanded teachers.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.23185 [cs.LG] |
| (or arXiv:2610.23185v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23185 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haoyu Chen [view email]
[v1]
Fri, 16 Oct 2026 09:23:22 UTC (874 KB)
来源：arXiv cs.AI · arxiv.org