arXiv cs.CL· Zeyu Leo Liu, Yang Li, Bo Pang, Emre Can Acikgoz, Xuan-Phi Nguyen, Anurag Koul, Srijan Bansal, Erik Nijkamp, Shafiq Joty, Semih Yavuz·· 11 小时前AI 评分38
拆解强到弱 OPD 与 GRPO 的计算效率对比
Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
AI 导读
系统对比 vanilla reverse-KL OPD 与 GRPO 后发现，使用外部后训练教师时 OPD 在单位训练 FLOP 的准确率上普遍不如 GRPO，且常出现回复长度膨胀。
正文 · 原文
Abstract:On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation. Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with $50\times$ fewer distillation FLOPs than GRPO needs to reach the same accuracy. Following the same teacher setting, we consistently observe this high efficiency with a larger student, on code, and in a second model family. On a single domain, however, the GRPO checkpoint is already the desired model, so distilling it adds to total compute. Across domains, separate GRPO checkpoints hold different capabilities, and no single checkpoint serves every domain. Multi-teacher OPD (DeepSeek-AI, 2026) consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs. Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the 2-domain mix with 42.0% fewer total FLOPs.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.08453 [cs.LG] |
| (or arXiv:2611.08453v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08453 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zeyu Liu [view email]
[v1]
Thu, 5 Nov 2026 18:47:05 UTC (1,463 KB)
来源：arXiv cs.CL · arxiv.org