arXiv cs.CL· Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron·· 13 小时前AI 评分38
REOPOLD：用松弛式在线策略蒸馏高效扩展推理能力
Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
AI 导读
研究者提出 REOPOLD（Relaxed On-Policy Distillation）框架，通过控制教师反馈的应用位置、强度和分配规则变化时机，缓解采样 token 在线策略蒸馏中的熵坍缩与负迁移问题。在多种推理任务上，该方法相比近期后训练基线最高提升 12 倍样本效率，并可扩展至跨词表蒸馏、自蒸馏与测试时扩展。
正文 · 原文
Abstract:On-policy distillation (OPD) is a promising approach for transferring reasoning capabilities to capacity-constrained models, yet sampled-token OPD often suffers from entropy collapse and negative transfer. We identify uniform credit allocation as a key bottleneck: existing objectives obtain dense teacher feedback on student-generated trajectories, but apply it uniformly across sampled tokens despite highly heterogeneous token-level signals. Most tokens carry redundant near-zero credit, while rare heavy-tailed negative credits can dominate updates and prematurely suppress plausible reasoning trajectories. We introduce REOPOLD (Relaxed On-Policy Distillation), a framework that relaxes this uniform assignment by controlling where teacher feedback is applied, how strongly it affects the update, and when the allocation rule changes during training. Across diverse reasoning tasks, REOPOLD improves sampled-token OPD over recent post-training baselines with up to 12x higher sample efficiency, and extends to cross-vocabulary distillation, self-distillation, and test-time scaling.
| Comments: | NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.11137 [cs.LG] |
| (or arXiv:2603.11137v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.11137 arXiv-issued DOI via DataCite |
Submission history
From: Jongwoo Ko [view email]
[v1]
Wed, 11 Mar 2026 16:26:52 UTC (1,228 KB)
[v2]
Tue, 27 Oct 2026 17:53:10 UTC (1,042 KB)
来源：arXiv cs.CL · arxiv.org