arXiv cs.AI· Shiping Yang, Chuanqi Tang, Hossein Hajimirsadeghi, Layla El Asri, Mohammad Amin Shabani, Angel X. Chang·· 14 小时前AI 评分31
拆解不确定性形状：On-Policy Distillation 学到了什么，以及如何预测
Distilling the Shape of Uncertainty: What On-Policy Distillation Learns and How to Predict It
AI 导读
研究揭示 on-policy distillation（OPD）不仅学习“如何分支”（考虑哪些候选 token），还学习“在哪里分支”（保持推理路径开放的位置），现有 token 重叠指标只捕捉前者。
正文 · 原文
Abstract:On-policy distillation (OPD) is increasingly adopted in the post-training of large language models. However, a stronger teacher may fail to improve the student. Prior work attributes this to a lack of new knowledge, but what that knowledge is remains unclear. We show that OPD learns not only \textit{how to branch}, i.e., which candidate tokens to consider, but also \textit{where to branch}, i.e., where to keep reasoning paths open. The existing token-overlap metric captures only the former. We introduce the entropy shape, the sequence of next-token entropies, to characterize the latter, and propose the Entropy Shape Correlation (ESC) to measure student-teacher agreement in entropy shape. Tracking ESC throughout training and intervening on a teacher's entropy while preserving its token preferences, we empirically show that \textit{where to branch} causally drives OPD gains, independently of \textit{how to branch}. Finally, we formulate OPD outcome prediction as a teacher selection task and establish an evaluation protocol for it, under which pre-OPD ESC selects the best teacher for both students and outperforms existing predictors. Code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39020 [cs.CL] |
| (or arXiv:2610.39020v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39020 arXiv-issued DOI via DataCite |
Submission history
From: Shiping Yang [view email]
[v1]
Wed, 28 Oct 2026 22:53:06 UTC (165 KB)
来源：arXiv cs.AI · arxiv.org