arXiv cs.CL· Guan Zhe Hong, Sumeet Ramesh Motwani, Liang Chen, Yangchen Pan, Junchi Yu, Jialin Yu, Philip Torr, Yi Ren·· 12 小时前AI 评分31
论文解析 on-policy 蒸馏的学习动力学
Learning Dynamics of On-Policy Distillation
AI 导读
一项机理性分析将 on-policy 蒸馏（OPD）的反馈回路拆为两个阶段：蒸馏信号转化为参数更新，以及更新通过广播改变各前缀预测并重塑 rollout 分布。
正文 · 原文
Abstract:On-policy distillation (OPD) is emerging as an important paradigm for language-model post-training, combining dense teacher supervision with training on student-generated trajectories. This creates a feedback loop where each update changes the student and thereby the training distribution that generates subsequent updates. Yet, how this feedback shapes learning and generalization remains poorly understood. We conduct a mechanistic analysis of OPD's learning dynamics, distinguishing the two feedback loop stages: conversion of the distillation signal into parameter updates, and broadcast through which these updates change predictions across prefixes and reshape the rollout distribution. This framework helps us understand both limited student responsiveness despite substantial teacher-student disagreement, and degeneration when updates that improve local teacher agreement also increase the probability of unsuccessful continuations, exposing the student to unreliable supervision in subsequent rollouts. Finally, we show that offline distillation can prepare a student for OPD without significantly improving immediate held-out accuracy. While OPD alone may degrade generation, it can yield substantial accuracy gains after sufficient offline preparation. Together, our results connect teacher supervision, student geometry, and the evolving rollout distribution in determining the learning dynamics of OPD and its effects on downstream performance.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.08016 [cs.LG] |
| (or arXiv:2611.08016v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08016 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guan Zhe Hong [view email]
[v1]
Thu, 5 Nov 2026 13:38:26 UTC (916 KB)
来源：arXiv cs.CL · arxiv.org