每枚硬币都有两面：大语言模型在线策略蒸馏中泛化性的双重本质
Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
一项受控研究发现，在线策略蒸馏（OPD）传递的是教师的推理行为而非具体答案，训练难度几乎无关，连教师从未解出的问题也有用。迁移效果强烈依赖师生同源关系：同源组合可跨语言、推理跨度乃至其他领域逼近教师，跨源组合则基本只拟合训练分布。这种广泛影响是双刃剑——按领域路由提示词无法限制各教师的影响，多教师组合会在其能力间形成依赖混合比例的跷跷板效应。
Authors:Zhaoyi Li, Deyang Kong, Yuan Wei, Evan Yang, Ranran Shen, Mahardika Krisna Ihsani, Ming Yang, Wei Zhang, Chuan Hao, Jian Yang, Ran Tao, Bryan Dai, Shikun Zhang, Wei Ye, Ying Wei, Defu Lian
Abstract:On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher's influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosing multi-teacher OPD.
| Comments: | Under Review |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.16647 [cs.CL] |
| (or arXiv:2608.16647v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.16647 arXiv-issued DOI via DataCite |
Submission history
From: Zhaoyi Li [view email]
[v1]
Mon, 17 Aug 2026 14:46:53 UTC (1,159 KB)
[v2]
Sun, 23 Aug 2026 15:08:29 UTC (1,183 KB)
[v3]
Fri, 23 Oct 2026 14:14:20 UTC (1,225 KB)
来源：arXiv cs.CL · arxiv.org