arXiv cs.AI· Shuo Yang, Zheyu Zhang, Mahito Sugiyama, Gjergji Kasneci·· 3 天前AI 评分32
当训练集中于学生失败题目时，On-Policy Distillation 为何反而强化错误
Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
AI 导读
研究用 Qwen-3 和 Phi-4 在统一竞赛数学协议下评测五种 On-Policy Distillation（OPD）机制，发现当训练仅集中于学生当前做错的题目时，OPD 无法提升准确率，八个配置全部下降，双尾符号检验 p=0.0078。相同流程在不集中于失败题目时准确率提升 4.6 个百分点，说明该失效取决于训练机制而非系统性缺陷。
正文 · 原文
Abstract:On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.
| Comments: | 10 pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18011 [cs.LG] |
| (or arXiv:2610.18011v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18011 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuo Yang [view email]
[v1]
Tue, 13 Oct 2026 09:10:58 UTC (318 KB)
来源：arXiv cs.AI · arxiv.org