arXiv cs.CL· Guangzeng Han, Yu Wu, Xiaolei Huang·· 12 小时前AI 评分38
Order-Only On-Policy Distillation：仅用教师排序做在线策略蒸馏的 O³PD 方法
Order-Only On-Policy Distillation
AI 导读
研究者提出 Order-Only On-Policy Distillation（O³PD），在每个学生生成的 prefix 上只让学生复现教师 top-K token 的排序，而不拟合其概率值，因此目标对教师温度不变、无需调温度。
正文 · 原文
Abstract:On-policy distillation (OPD) trains a student on its own generations by matching the teacher's token probabilities. These probabilities entangle two kinds of information: which tokens the teacher prefers, and how confident it is. The latter reflects the teacher's own capacity and calibration, which a smaller student may be unable to reproduce. Because the teacher temperature changes these values without changing the ordering, OPD is also sensitive to the temperature choice. We ask whether the teacher's ordering of candidate tokens is enough, and propose Order-Only On-Policy Distillation (O$^{3}$PD). At each student-generated prefix, O$^{3}$PD trains the student to reproduce the order of the teacher's top-$K$ tokens, without fitting their probability values. The objective is therefore invariant to the teacher temperature and needs no temperature tuning. Across six teacher--student pairs, O$^{3}$PD outperforms OPD on the average over HMMT25, AIME24, and AIME25 in every pair, by up to 2.9 points, with gains of up to 5.6 points on AIME25. Other ranking losses yield similar gains, so the benefit comes from ordinal supervision itself. Analysis shows that O$^{3}$PD departs further from the teacher's probabilities than OPD, yet reproduces its ranking more faithfully and yields a stronger student. The ranking, not the probability values, thus carries the knowledge that on-policy distillation needs to transfer.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.02112 [cs.LG] |
| (or arXiv:2611.02112v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02112 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guangzeng Han [view email]
[v1]
Sun, 1 Nov 2026 18:38:29 UTC (648 KB)
来源：arXiv cs.CL · arxiv.org