arXiv cs.CL· Runyang You, Yongqi Li, Tiezheng Yu, Wenjie Li·· 12 小时前AI 评分31
On-Policy Transition Distillation：用层间预测转移增强在线策略蒸馏
On-Policy Transition Distillation
AI 导读
研究提出 On-Policy Transition Distillation（OPTD），在在线策略蒸馏（OPD）匹配师生最终层分布的基础上，额外对齐学生生成 rollout 上师生逐层 log 概率增量。
正文 · 原文
Abstract:On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts. We investigate whether depthwise prediction transitions, the changes in predictions between layers, can provide additional task-relevant supervision. Our diagnostic study finds that stronger teacher--student agreement in intermediate transitions is associated with higher answer correctness within questions, with correct responses exhibiting lower mean transition KL across all measured intermediate layer pairs. These observations motivate On-Policy Transition Distillation (OPTD), which augments final-layer distribution matching by aligning teacher and student layer-to-layer log-probability increments on student-generated rollouts. Evaluations across three backbones and eight benchmarks spanning mathematics, coding, and instruction following show that OPTD achieves the best or tied-best overall performance in both dual- and three-domain distillation. In the three-domain setting, OPTD recovers $\approx 90\%$ of the overall teacher--student capability gap, compared with $\approx 75\%$ for OPD.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.28048 [cs.CL] |
| (or arXiv:2610.28048v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28048 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Runyang You [view email]
[v1]
Tue, 20 Oct 2026 11:37:31 UTC (1,388 KB)
来源：arXiv cs.CL · arxiv.org