arXiv cs.AI· Liangyao Jin, Zhixin Wang, Liming Liu, Mingjun Zhang, Zhengzhou Zhu, Fangcheng Fu, Jinlong Hou, Yuan Cheng, Xiaohe Hu, Ping Zhang, Siyuan Feng·· 2 天前AI 评分22
Router Drift Control：缓解 MoE 强化学习中的离策略退化
Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning
AI 导读
针对 MoE 模型强化学习中 rollout 与训练路由不一致导致的 router drift，研究者提出 Router Drift Control（RDC），以路由分歧作为离策略陈旧度信号，在序列级别过滤高漂移响应，避免 R3 路由重放带来的梯度偏差。
正文 · 原文
Authors:Liangyao Jin, Zhixin Wang, Liming Liu, Mingjun Zhang, Zhengzhou Zhu, Fangcheng Fu, Jinlong Hou, Yuan Cheng, Xiaohe Hu, Ping Zhang, Siyuan Feng
Abstract:Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous this http URL resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout--training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows that routing replay introduces gradient bias by losing contributions from experts selected by the training router but not the rollout router. To address this, we propose Router Drift Control (RDC), which uses routing disagreement between rollout and training as a signal of off-policy staleness and selectively rejects high-drift responses. The sequence-level filtering avoids gradient bias introduced by replay while preserving native expert selection and the associated learning signals for retained responses. Experiments show that RDC achieves larger gains as policy staleness increases and remains effective in asynchronous RL. In asynchronous training, RDC combined with Truncated Importance Sampling (TIS) improves the average peak validation accuracy across AIME 2024 and AIME 2025 by 5.5 points over TIS and 4.5 points over R3.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.21682 [cs.LG] |
| (or arXiv:2610.21682v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21682 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhixn Wang [view email]
[v1]
Thu, 15 Oct 2026 11:09:10 UTC (562 KB)
来源：arXiv cs.AI · arxiv.org