arXiv cs.AI· Yutong Chen, Guangfu Guo, Zhichao Xu, Kunpeng Liu·· 3 天前AI 评分25
DualOPSD：在 On-Policy 自蒸馏中闭合反馈回路
DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation
AI 导读
DualOPSD 通过让特权教师模型在学生每次更新后同步适应，闭合了 OPSD 中教师收不到学生反馈的回路，并用基于冻结初始模型的残差目标保留特权上下文效果。在 Qwen3-8B/4B/1.7B 和三个数学基准上，DualOPSD 的 avg@12 准确率较 OPSD 提升 9.83 个百分点。诊断分析显示其双向 KL 更低、失配更少，代码已开源。
正文 · 原文
Abstract:On-policy self-distillation (OPSD) combines student generated trajectories with dense supervision from a privileged teacher of the same model. Its pointwise clipping protects the student from locally incompatible teacher directions, but the fixed teacher receives no feedback about the supervision the student rejects, which limits effective learning from the teacher. We introduce DualOPSD, which alternates updates in a coupled teacher-student system. The student retains OPSD's clipped forward KL objective. The privileged teacher adapts to the updated student after each student step. A residual target based on the frozen initial model preserves the effect of privileged context during teacher adaptation. Across Qwen3-8B/4B/1.7B and three math benchmarks, DualOPSD improves avg@12 accuracy over OPSD by 9.83 percentage points. Further diagnostic analyses show lower bidirectional KL, less persistent mismatch, and retained sensitivity to privileged reference. Together, these results demonstrate that DualOPSD can be a more effective form of privileged self-distillation. Code is available at: this https URL.
| Comments: | preprint |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.26019 [cs.LG] |
| (or arXiv:2608.26019v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.26019 arXiv-issued DOI via DataCite |
Submission history
From: Yutong Chen [view email]
[v1]
Wed, 26 Aug 2026 17:01:21 UTC (171 KB)
[v2]
Tue, 13 Oct 2026 17:08:56 UTC (356 KB)
来源：arXiv cs.AI · arxiv.org