arXiv cs.CL· Taiming Lu, Zhuang Liu·· 13 小时前AI 评分34
强教师并非必需？论 LLM 预训练中的知识蒸馏
Strong Teacher Not Needed? On Distillation in LLM Pretraining
AI 导读
一项 NeurIPS 2026 研究挑战了知识蒸馏必须“强教师带弱学生”的假设：在 LLM 预训练中，只要合理混合语言建模损失与蒸馏损失，即使规模小、训练不足的教师也能提升更大学生的表现。而通过增加参数量或训练 token 继续强化教师，蒸馏收益可能饱和甚至反转。研究还发现，蒸馏对分布外和下游性能等泛化能力的提升比对域内拟合更明显。
正文 · 原文
Abstract:Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in large language model pretraining. By varying architecture sizes and training token budgets, we create strong-to-weak, same-level, and weak-to-strong teacher-student relationships, and study distillation's effectiveness under each. We find that the teacher need not be strong: with proper mixing of the language modeling and knowledge distillation losses, even small and undertrained teachers improve larger students. At the same time, a stronger teacher is not always better: pushing the teacher further, through more parameters or more training tokens, can saturate or even reverse the distillation gains. We further observe that distillation improves generalization (out-of-distribution and downstream performance) more readily than in-domain fitting. Together, these results challenge the common belief that distillation pretraining always requires a strong teacher.
| Comments: | NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.23857 [cs.LG] |
| (or arXiv:2605.23857v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.23857 arXiv-issued DOI via DataCite |
Submission history
From: TaiMing Lu [view email]
[v1]
Fri, 22 May 2026 17:16:35 UTC (5,513 KB)
[v2]
Mon, 26 Oct 2026 22:40:35 UTC (5,037 KB)
来源：arXiv cs.CL · arxiv.org