arXiv cs.AI· Shripad V. Deshmukh, Christian Bock, Kangrui Ruan, Abhijeet Awasthi, Vijay Lingam, Rein Houthooft, Manish Prajapat·· 12 小时前AI 评分38
Projected Distillation：用最小修正把失败尝试变成正确目标，实现近同策略学习
Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets
AI 导读
研究者提出 projected distillation，将学生模型的失败尝试最小幅度编辑为正确目标，并用对比更新偏好修正、随学生演化刷新，从而在学生全部尝试失败时仍能进行近同策略学习。
正文 · 原文
Abstract:On-policy learning is central to post-training: training on a model's own generations limits distribution shift and helps preserve existing capabilities. On hard problems, however, on-policy RL provides no reward-based learning signal when all attempts fail. On-policy distillation provides next-token supervision from a teacher, but on hard problems, if a rollout goes wrong early, subsequent supervision is conditioned on prefixes that already contain the mistake. Correct behavior can instead come from a gold or teacher solution, but such off-policy targets can differ substantially from the model's generations and, as we show, may erode existing capabilities. We therefore treat on-policyness as a continuum and propose projected distillation, which minimally edits the student's failed attempts into correct targets. The method uses contrastive updates to favor corrections over failed attempts and refreshes these corrections as the student evolves, enabling near-on-policy learning even when all student attempts fail. Theoretically, we prove that the student's learning regret grows with the edit size: it scales linearly with the horizon under minimal correction, as in DAgger, and quadratically under full-answer replacement, as in behavior cloning. Empirically, across two students (Qwen3-8B and OLMo-2-7B) and five domains spanning math, code, science, and tool use, projected distillation gives the highest area under the specialization-retention frontier (best on 8 of 9 cells against GRPO and OPD, and 7 of 9 against SPOT and SODA). Our ablations highlight the importance of online corrections, the contrastive loss, and on-policyness: moving targets farther off-policy steadily worsens the specialization-retention trade-off, with retention falling from 0.916 to 0.744 when minimal corrections are replaced by student-agnostic gold solutions.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.27467 [cs.LG] |
| (or arXiv:2610.27467v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27467 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shripad Deshmukh [view email]
[v1]
Tue, 20 Oct 2026 01:38:01 UTC (574 KB)
来源：arXiv cs.AI · arxiv.org