arXiv cs.CL· Xue Zhang, Songming Zhang, Yunlong Liang, Fandong Meng, Yufeng Chen, Jinan Xu·· 11 小时前AI 评分26
SPTA：让学生前缀适配教师，提升在线蒸馏可靠性
Meet Students Where They Are: Student-Prefix Teacher Adaptation for On-Policy Distillation
AI 导读
针对在线蒸馏中学生生成前缀含推理错误、导致教师 token 级监督失准的问题，研究提出 Student-Prefix Teacher Adaptation（SPTA）：在 OPD 开始前收集随机长度的学生 rollout 前缀，用 RLVR 持续训练教师从这些前缀续写，使其学会从学生引入的状态中恢复而非模仿学生推理。
正文 · 原文
Abstract:On-policy distillation (OPD) mitigates the student's exposure bias by optimizing on student-generated trajectories. However, student-generated prefixes may contain reasoning errors or follow patterns unfamiliar to the teacher, undermining the reliability of its token-level supervision. This degradation becomes more pronounced as prefixes grow longer. To address this teacher-side distribution mismatch, we propose Student-Prefix Teacher Adaptation (SPTA), which adapts the teacher to the states on which it is queried during distillation. Specifically, before OPD starts, we collect student rollout prefixes with random lengths and continually train the teacher to continue from the prefixes using reinforcement learning with verifiable rewards (RLVR). By treating each prefix as context and rewarding correct final answers, the teacher learns to recover from student-induced states rather than imitate the student's reasoning. Then, the adapted teacher is used for standard OPD, with no changes to the distillation objective. Experiments on mathematical reasoning benchmarks show that our SPTA improves average student performance over standard OPD and OPD with task-level RLVR-adapted teachers. These results highlight the importance of adapting teachers not only to solve problems independently, but also to provide reliable guidance on the states their students actually visit.
| Comments: | 15 pages |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.03133 [cs.CL] |
| (or arXiv:2611.03133v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.03133 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xue Zhang [view email]
[v1]
Mon, 2 Nov 2026 13:45:23 UTC (311 KB)
来源：arXiv cs.CL · arxiv.org