arXiv cs.AI· Yunjie Chen, Zihao Chen, Xiaoxin Chen, Fan Wang·· 3 天前AI 评分29
REGEN：用离线强化学习回收回放记忆实现专家到通才蒸馏
REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning
AI 导读
REGEN 提出用离线强化学习回收教师模型专项 RL 训练产生的回放记忆，训练通才模型，从而完全解耦 rollout 采样与反向传播。在数学推理、代码生成和指令遵循任务上，REGEN 以显著更低的成本达到多教师在线蒸馏（MOPD）的准确率。该方法有望将在线 RL 从一次性学习阶段转变为数据合成过程，代码已开源。
正文 · 原文
Abstract:Large-scale online reinforcement learning (RL) is the predominant means of eliciting advanced abilities including long-term reasoning and agentic tool use in large language models (LLMs). However, continuing to scale it across vast task domains of interest remains challenging in both computational infrastructure and cost, especially when considering RL as merely a one-off learning stage. Recently, a widely used technique for distilling knowledge across various domains and training stages, multi-teacher on-policy distillation (MOPD), helps to decouple the RL stage, saving costs, while maintaining generality across vast domains. Nonetheless, similar to online RL, MOPD requires coupled inference and backward passes, which continues to limit its scalability and computational efficiency. To address these challenges, we propose REGEN: Replay-recycling for Expert-to-Generalist Distillation with Offline RL. Instead of distilling from multiple teacher models, REGEN trains a generalist by simply recycling the replay memory -- the free by-product of the teachers' specialized RL training -- and employing offline RL algorithms. REGEN completely decouples the rollout sampling from the backward training process and thus greatly reduces the training cost. Across mathematical reasoning, code generation, and instruction following, REGEN matches the accuracy of MOPD at substantially lower cost. It potentially turns online RL into a data synthesis process instead of a one-off learning stage, and can be extended to large-scale post-training without requiring heavy computational load. Code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.19450 [cs.LG] |
| (or arXiv:2607.19450v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.19450 arXiv-issued DOI via DataCite |
Submission history
From: Yunjie Chen [view email]
[v1]
Tue, 21 Jul 2026 13:36:23 UTC (1,425 KB)
[v2]
Wed, 29 Jul 2026 12:12:34 UTC (1,425 KB)
[v3]
Sat, 12 Sep 2026 15:22:15 UTC (430 KB)
[v4]
Mon, 12 Oct 2026 14:09:54 UTC (430 KB)
来源：arXiv cs.AI · arxiv.org