arXiv cs.AI· Yue Wu, Xueyu Zhou, Yangrong Hu, Jian Huang·· 10 小时前AI 评分26
Bootstrapped On-Policy Distillation（BOPD）：两阶段框架解决 OPSD 持续训练性能退化问题
Bootstrapped On-Policy Distillation
AI 导读
研究者提出 Bootstrapped On-Policy Distillation（BOPD），一个两阶段框架：先用 OPSD 将参考条件监督中的任务相关信息内化到中间模型，再把该无参考中间模型蒸馏到初始模型的新副本中。
正文 · 原文
Abstract:On-policy distillation (OPD) improves language models by distilling teacher predictions on student-generated trajectories, while on-policy self-distillation (OPSD) removes the need for an external teacher by using a reference-conditioned copy of the model as the teacher. Despite its great early improvements, we find that OPSD does not remain consistently effective under continued training and can eventually suffer substantial performance degradation. We study this training behavior and link the degradation to a growing mismatch between the evolving reference-free student and the frozen privileged teacher. In the early stage of OPSD, the privileged teacher remains closely aligned with the student and provides useful guidance for improving the student's generation trajectories. However, as training progresses, the growing distributional mismatch between the privileged teacher and reference-free student can cause the teacher to provide misleading supervision, ultimately degrading student performance. This phenomenon motivates us to first leverage privileged reference supervision to acquire useful knowledge and then transfer it to a fresh student. We introduce Bootstrapped On-Policy Distillation (BOPD), a two-stage framework that first uses OPSD to internalize task-relevant information from reference-conditioned supervision into an intermediate model, and then distills this reference-free intermediate model into a fresh copy of the initial model. Experiments across models show that BOPD improves mathematical reasoning and code generation beyond both the initial models and the constructed OPSD teachers. These results suggest that privileged reference supervision can be more effectively exploited by first transferring its task-relevant signal into a task-adapted model and then distilling from that model without reference conditioning.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13680 [cs.LG] |
| (or arXiv:2611.13680v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13680 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yue Wu [view email]
[v1]
Tue, 10 Nov 2026 07:30:34 UTC (254 KB)
来源：arXiv cs.AI · arxiv.org