arXiv cs.CL· Zhenhe Wu, Jian Yang, Yaping Jin, Xinlai Kang, Jie Hu, Hanting Chen·· 3 天前AI 评分20
PA1-OPD：面向语言模型预训练的一步前缀增强同策略蒸馏
PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretraining
AI 导读
PA1-OPD 在语料前缀知识蒸馏基础上，对每个选定前缀增加一次学生采样转移后的教师反馈，从而扩展监督状态并保留下一 token 预测。
正文 · 原文
Abstract:Knowledge distillation (KD) improves language-model pretraining by matching teacher distributions at corpus prefixes, but provides no direct supervision at contexts induced by student predictions. We introduce PA1-OPD, which augments corpus-path distillation with teacher feedback after one student-sampled transition per selected prefix. This expands the supervised states while retaining next-token prediction and corpus-path KD. Data-anchored roots support parallel sampling, and differentiable tree attention shares prefix computation without changing inference. In two continued-pretraining settings, full coverage improves held-out cross-entropy over KD across C4, PG-19, and Pile at every evaluated checkpoint. In Stage2, selecting the 10% of roots with highest student surprisal adds 48.85% to KD's Macro cross-entropy gain over next-token training. Gains remain when branch supervision excludes successor states already on the corpus path, supporting the value of feedback beyond observed contexts. Additional depth or breadth improves aggregate results at unchanged per-term supervision weight, with benefits varying by corpus. These findings support local student-induced supervision as a complement to pretraining distillation.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19601 [cs.CL] |
| (or arXiv:2610.19601v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19601 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhenhe Wu [view email]
[v1]
Wed, 14 Oct 2026 05:23:00 UTC (213 KB)
来源：arXiv cs.CL · arxiv.org