arXiv cs.CL· Bowen Fang (Columbia University), Dingmin Wang (AWS AI Lab), Hengzhi Pei (Amazon AGI)·· 11 小时前AI 评分26
D²-OPD：在推演式在线策略蒸馏中分离方向与距离
$D^2$-OPD: Separating Direction and Distance in Extrapolative On-Policy Distillation
AI 导读
D²-OPD 提出将推演式在线策略蒸馏（OPD）中的外推方向与外推距离解耦：用逐前缀的硬 KL 约束限制目标偏离教师模型的距离，由教师模型后训练偏移与初始学生-教师差距的自适应组合决定外推方向，并以正向 KL 优化学生模型。在数学推理基准的三类设置中，D²-OPD 均取得最高分，最强基线最多被超出 9.45 分。
正文 · 原文
Abstract:On-policy distillation (OPD) trains a student model to match a teacher's next-token distributions on student-generated prefixes. Extrapolative OPD further extends the teacher's log-probability shift relative to a reference policy to construct stronger learning targets. However, existing methods couple extrapolation direction and distance, allowing the magnitude of the teacher's shift to dictate how far the target moves. We propose $D^2$-OPD, decoupling these two factors: a per-prefix hard KL constraint bounds the target's distance from the teacher, while an adaptive combination of the teacher's post-training shift and the initial student--teacher gap determines the extrapolation direction. We optimize the student toward this target using forward KL, providing learning signals even for low-probability tokens. We evaluate three settings: where standard OPD performs well, where it struggles with base-model initialization, and where no teacher reference is available. On mathematical reasoning benchmarks, $D^2$-OPD attains the highest score in all three settings, outperforming the strongest baseline by up to 9.45 points. Ablations validate the contributions of each component.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.07241 [cs.LG] |
| (or arXiv:2611.07241v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07241 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bowen Fang [view email]
[v1]
Wed, 4 Nov 2026 23:45:46 UTC (341 KB)
来源：arXiv cs.CL · arxiv.org