arXiv cs.AI· Zhepei Hong, Yukai Wang, Mao Zheng, Mingyang Song, Gengsheng Li, Houcheng Jiang, Dan Zhang, Haiyun Guo, Junfeng Fang, Tat-Seng Chua·· 10 小时前AI 评分36
基于评分标准的黑盒在线策略蒸馏 ROPD：用文本反馈替代教师 logits
Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
AI 导读
研究提出 ROPD，通过对比教师回答与学生当前 rollout 生成评分标准（rubric），为黑盒在线策略蒸馏提供纯文本反馈接口。在四个竞赛数学基准上，纯文本 ROPD 优于基于 logits 的 OPD，以 9.6 倍更少的训练样本达到标准 logit-based OPD 的最佳 AIME24 准确率。
正文 · 原文
Abstract:On-policy distillation (OPD) is a powerful paradigm for LLM post-training, yet its reliance on teacher logits restricts it to white-box teachers. In this paper, we study whether rubric-based supervision can provide a text-only feedback interface for black-box OPD, and what makes it effective. Our instantiation, Rubric-based On-Policy Distillation (ROPD), induces rubrics by contrasting teacher responses with the student's current rollouts and uses them to score these rollouts for on-policy optimization. We find that, under matched settings, text-only ROPD outperforms logit-based OPD methods on four competition-math benchmarks, reaching the best AIME24 accuracy of standard logit-based OPD with 9.6x fewer training samples. Our analysis yields three findings: rubric rewards track correctness far better than teacher log-likelihood (AUC 0.90 vs. 0.35); training with them repairs more failed criteria and regresses fewer passed ones than logit-based supervision; and contrasting teacher responses with student rollouts is central to rubric quality. These results position rubric-based OPD as a flexible, black-box-compatible alternative to logit-based OPD. Our implementation is available at this https URL.
| Comments: | Preprint. Code is available at this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.07396 [cs.LG] |
| (or arXiv:2605.07396v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07396 arXiv-issued DOI via DataCite |
Submission history
From: Zhepei Hong [view email]
[v1]
Fri, 8 May 2026 07:52:15 UTC (4,254 KB)
[v2]
Tue, 10 Nov 2026 03:40:16 UTC (872 KB)
来源：arXiv cs.AI · arxiv.org