arXiv cs.AI· Guangyu Liu, Siyuan Gan, Shilong Zhou, Linjian Meng, Boyan Wang, Jing Huo·· 12 小时前AI 评分22
Adaptive Pullback：在策略蒸馏中于惩罚跃变处重新加权监督
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation
AI 导读
针对在策略蒸馏（OPD）中正 log-ratio 惩罚无法区分持续高惩罚与突增的问题，研究者提出 AP-OPD，在检测到惩罚跃变时以前一 token 块为局部参考，统一缩放后续块的监督系数，使加权平均惩罚与前一块的原始均值对齐，无需额外强度常数。该方法在数学推理与代码生成任务上均取得性能提升。
正文 · 原文
Abstract:On-policy distillation (OPD) supervises student-generated responses using sampled-token student-teacher log-ratios. A positive log-ratio penalizes a token that the student assigns higher probability to than the teacher, but its current value does not distinguish sustained high penalties from abrupt increases along a response. We examine these local changes by comparing consecutive blocks of tokens within the same response. Our diagnostic identifies penalty jumps, abrupt increases in average penalty that persist on held-out tokens. Across these jumps, penalties rise while teacher confidence decreases. We call this pattern penalty-confidence tension. Boundaries into high-penalty blocks do not exhibit comparable confidence changes, showing that local changes provide information beyond the current penalty value. Motivated by this observation, we propose Adaptive Pullback OPD (AP-OPD). At each detected jump, AP-OPD uses the preceding block as a local reference and uniformly scales all supervision coefficients in the following block so that its weighted average penalty matches the preceding block's original mean. This reference determines the scaling factor without an additional strength constant. AP-OPD improves performance on both mathematical reasoning and code generation.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.28194 [cs.LG] |
| (or arXiv:2610.28194v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28194 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Siyuan Gan [view email]
[v1]
Tue, 20 Oct 2026 13:34:43 UTC (194 KB)
来源：arXiv cs.AI · arxiv.org