arXiv cs.CL· Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang·· 2 天前AI 评分22
Waypoint-OPD：通过门控同策略蒸馏迁移 RL 引发的策略偏移
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
AI 导读
Waypoint-OPD 通过引入同一 RL 训练的中间教师检查点，比较早期与晚期策略偏移在动作偏好方向上的一致性，并据此对端点偏移进行门控，方向相悖时予以抑制。该方法仅需额外一次冻结教师打分，在数学与代码推理的六种领域-学生组合中，以 Qwen3-4B 为教师、Qwen3-8B/14B/30B-A3B 为学生，均取得最高领域平均分。Qwen3-8B 消融显示，移除或打乱门控会降低数学推理表现。
正文 · 原文
Abstract:Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.21601 [cs.LG] |
| (or arXiv:2610.21601v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21601 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guochao Jiang [view email]
[v1]
Thu, 15 Oct 2026 09:47:19 UTC (945 KB)
来源：arXiv cs.CL · arxiv.org