arXiv cs.AI· Yingping Wang, Min Cai, Chenyu Yuan, Xi Ye, Zhijiang Guo, Jiayi Huang·· 17 小时前AI 评分34
DoubleFree：利用探索历史实现无 rollout 的多任务 LLM 训练
DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training
AI 导读
DoubleFree 复用领域 RL 训练中产生的历史轨迹完成多任务整合，无需重新生成 on-policy 响应，并预计算教师监督以避免在线教师开销。在 Qwen3-4B、Qwen3-8B 和 SmolLM3-3B 的数学、代码与指令遵循任务上，该方法实现 2.99-5.75 倍整合加速，GPU 小时数减少 80.9%-90.1%，多领域性能保持相当。
正文 · 原文
Abstract:Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves $2.99$-$5.75\times$ consolidation speedups and reduces GPU-hours by $80.9$-$90.1\%$, while maintaining comparable multi-domain performance.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34272 [cs.LG] |
| (or arXiv:2610.34272v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34272 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingping Wang [view email]
[v1]
Sun, 25 Oct 2026 13:03:18 UTC (439 KB)
来源：arXiv cs.AI · arxiv.org