arXiv cs.AI· Zhenrui Yue, Gongxiaohui Chen, Huimin Zeng, Honglei Zhuang, Jiashuo Sun, Zhen Han, Dong Wang·· 10 小时前AI 评分38
无需重采样:长程 RL 专家的离策略整合方法 MRPD
Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts
AI 导读
研究者提出多专家回放策略蒸馏(MRPD),直接回放各领域 RL 专家的训练日志来整合模型,无需重新采样或调用环境与奖励模型,整合成本仅为训练这些专家的约七分之一。在推理、搜索和 SQL 三个长程智能体任务上,MRPD 的学生模型大幅优于在线 RL 基线,并以远低于 MOPD 的成本超过同一批专家经 MOPD 整合的结果。分析显示回放轨迹贡献了大部分收益,且日志顺序比数量与奖励更重要。
正文 · 原文
Abstract:Frontier post-training follows a common recipe: train one expert per domain with reinforcement learning (RL), then consolidate the experts into one model via multi-teacher on-policy distillation (MOPD). Such consolidation is expensive: the student samples in every environment again, an expert scores every sampled token, and the student inherits the drift of the final experts. Yet we argue that the experts' RL runs have already produced what consolidation needs: rollouts with their sampling probabilities and rewards, in the order the experts learned from them. We propose multi-expert replay policy distillation (MRPD), which replays these training logs through the student in the experts' order. At each step, the policy gradient is recomputed on these scored trajectories, and the final experts' distributions are distilled at the logged prefixes. Following the experts' own trajectory reduces the training-inference mismatch and keeps the student within the clip range, so the replay is off-policy in origin but quasi-on-policy throughout consolidation. MRPD samples nothing and calls no environment or reward model: the trainer is the sole process, and consolidating the experts costs only about a seventh of the compute used to train them. On three long-horizon agentic domains (reasoning, search and SQL), MRPD yields the best student model, substantially outperforming online RL baselines, and surpassing MOPD from the same experts at a fraction of MOPD's cost. Our analysis on the same logs shows that the replayed trajectories carry most of the gain, and that both the policy gradient and the distillation term add to it. Most interestingly, the order of the logs matters more than their amount or rewards, suggesting that expert training logs are a curriculum, not a data mixture.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24059 [cs.LG] |
| (or arXiv:2611.24059v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24059 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhenrui Yue [view email]
[v1]
Wed, 18 Nov 2026 08:51:39 UTC (568 KB)
来源:arXiv cs.AI · arxiv.org