arXiv cs.CL· Mingyang Song, Mao Zheng·· 13 小时前AI 评分27
大语言模型在线策略蒸馏（On-Policy Distillation）综述
A Survey of On-Policy Distillation for Large Language Models
AI 导读
一篇综述提出以学生生成状态、教师信息与策略更新三者互动为统一框架，梳理大语言模型在线策略蒸馏（OPD）——即在学生模型自生成轨迹上施加教师指导的训练方法。该框架将蒸馏基础与自蒸馏、多教师集成、强化学习及交互式智能体相连接，并分析这些设置如何改变学生可获得的监督信号，进而关联能力迁移、训练效率与部署。综述聚焦学生演化过程中指导何时仍可靠且可学习，兼顾即时性能与后续学习能力。
正文 · 原文
Abstract:On-Policy Distillation (OPD) trains large language models with teacher guidance on student-generated trajectories, addressing the mismatch between fixed demonstrations and the histories encountered during deployment. As its use expands across language model training, different formulations of supervision and optimization make it difficult to compare methods and understand when they are effective. This survey develops a unified view of OPD through the interplay between student-generated states, teacher information, and policy updates. This framework connects the foundations of distillation with self-distillation, multi-teacher integration, reinforcement learning, and interactive agents. We examine how these settings change the supervision available to the student and relate the resulting design choices to capability transfer, training efficiency, and deployment. Our synthesis centers on the conditions under which guidance remains reliable and learnable as the student evolves, considering both immediate performance and the capacity for subsequent learning. We conclude by discussing open challenges in adapting supervision and allocating computation to support effective learning.
| Comments: | Work in Progress |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.00626 [cs.LG] |
| (or arXiv:2604.00626v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.00626 arXiv-issued DOI via DataCite |
Submission history
From: Mingyang Song [view email]
[v1]
Wed, 1 Apr 2026 08:32:34 UTC (84 KB)
[v2]
Tue, 12 May 2026 12:53:33 UTC (137 KB)
[v3]
Mon, 18 May 2026 12:04:15 UTC (152 KB)
[v4]
Thu, 18 Jun 2026 17:00:51 UTC (170 KB)
[v5]
Wed, 21 Oct 2026 08:22:26 UTC (579 KB)
来源：arXiv cs.CL · arxiv.org