arXiv cs.CL· Mayug Maniparambil, Ayush Rai, Ekansh Verma, Noel E. O'Connor, Fergal Reid·· 6 天前AI 评分31
只会打分的特权教师不会搜索：面向推理的批判条件化在线自蒸馏
A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
AI 导读
研究发现在线自蒸馏中，基于参考解打分的教师信号既无法定位学生 rollout 中的错误，也无法将概率推离错误 token，甚至难以区分同一问题下的正确与错误 rollout。
正文 · 原文
Abstract:On-policy self-distillation promises to replace the sparse outcome signal of verifiable-reward RL with dense token-level credit: a teacher conditioned on the reference solution scores the student's own rollout, and the resulting distributional shift is distilled back into the policy. We investigate whether this scoring signal actually contains corrective credit. Using gold error spans in student rollouts, we test the two signals consumed by self-distillation objectives, the full-distribution KL and the token log-ratio, for two properties: whether they localize errors and whether they push probability away from erroneous student tokens. They exhibit neither of these properties nor can they reliably distinguish correct from incorrect rollouts within the same problem. We argue that this failure is structural: locating an error in a reasoning trace is itself a search problem, while a scoring teacher is never explicitly asked to perform that search. The same model, with the same privileged information, localizes errors substantially better when allowed to generate a critique. This forms the motivation for \emph{critique-conditioned self-distillation}: first generate a critique of the current rollout, then condition the scoring teacher on that critique. The resulting signal localizes errors, is corrective within error spans, and shifts probability toward tokens that initiate backtracking such as ``Wait'', ``But'', and ``Actually''. We further show that the critique is useful both at test time, where resuming a wrong rollout with the critique in context converts math problems the model cannot solve by sampling, and in cold-start GRPO, where it provides an advantage signal on prompts for which verifiable rewards alone provide none.
| Comments: | Accepted as an oral at the FLLMPT workshop, NeurIPS 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.13744 [cs.AI] |
| (or arXiv:2610.13744v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13744 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mayug Maniparambil [view email]
[v1]
Fri, 9 Oct 2026 15:36:47 UTC (262 KB)
来源：arXiv cs.CL · arxiv.org