跳到正文
arXiv cs.AI· Hubert Strauss, Liam Fowl, Noam Razin·· 10 小时前AI 评分26

哪些 token、以何顺序、多快?On-Policy 蒸馏目标的理论分析

Which Tokens, in What Order, and How Fast? A Theoretical Analysis of On-Policy Distillation Objectives

AI 导读

该研究从理论上分析了 on-policy 蒸馏(OPD)目标的选择如何决定 token 优先级、学习顺序与优化效率。序列级 OPD 存在"未来一致性偏差",会先匹配教师模型靠后的 token 位置,而 token 级目标可避免这一反向学习顺序;前向 KL 散度优化景观良好,反向 KL 等其他 f-散度即使师生差距大也可能梯度消失。

正文 · 原文

View PDF HTML (experimental)

Abstract:On-policy distillation (OPD) trains a student language model to match a teacher model based on its own generations. When using OPD, one must choose whether to apply it at the sequence or token level and which divergence to minimize. However, despite the wide adoption of OPD in practice, the formal understanding of which OPD objective is appropriate for a given setting is limited. Toward addressing this gap, we theoretically analyze how the choice of OPD objective governs which tokens are prioritized, in what order, and how efficiently. Our analysis reveals that sequence-level OPD exhibits a future agreement bias, which myopic token-level objectives avoid. This bias induces a reverse learning order, in which sequence-level OPD first matches the teacher at later token positions, while at earlier positions it can favor tokens that are improbable under the teacher, and thus often undesirable. As for the choice of divergence, we prove that it strongly influences the amenability of OPD to gradient-based optimization. In particular, while the forward KL divergence enjoys a well-behaved objective landscape, other $f$-divergences, such as reverse KL, can suffer from vanishing gradients even when the student and teacher are far apart. Across language model post-training settings, we demonstrate the implications of these theoretical findings for which OPD objective is preferable, highlighting the importance of accounting for the interplay between the student, teacher, and task at hand.
Comments: Code available at this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
Cite as: arXiv:2611.23397 [cs.LG]
  (or arXiv:2611.23397v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.23397

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Noam Razin [view email]
[v1] Tue, 17 Nov 2026 19:01:02 UTC (379 KB)

来源:arXiv cs.AI · arxiv.org