arXiv cs.AI· Wenxuan Luo, Chenxi Li, Haoyue Liu, Yongxin Guo, Xiaoying Tang·· 2 天前AI 评分22
CCE-OPD：面向语言模型在线策略蒸馏的条件对比外推方法
Condition-Contrastive Extrapolation for On-Policy Distillation of Language Models
AI 导读
针对在线策略蒸馏（OPD）中教师模型易偏好违反问题前提的无效续写、进而鼓励错误推理步骤的问题，研究者提出 CCE-OPD，通过固定前缀对比与有界对数概率对比让教师监督对问题条件敏感。
正文 · 原文
Abstract:On-policy distillation (OPD) improves a student language model by training on its own generated trajectories with supervision from a stronger teacher. However, when conditioned on student-generated prefixes, the teacher can favor invalid continuations that violate the question's premises, causing OPD to encourage erroneous reasoning steps. As a remedy, we introduce Condition-Contrastive Extrapolated On-Policy Distillation (CCE-OPD), which makes teacher supervision sensitive to question conditions. CCE-OPD incorporates (1) a fixed-prefix comparison in which the same frozen teacher evaluates the complete question and a counterpart with a selected condition masked, and (2) a bounded log-probability contrast that reweights the teacher distribution into an auxiliary distillation target. On 200 diagnostic reasoning states, condition contrast reduces the rate of preferring invalid continuations from 27.0% to 16.0%. Extensive experiments across six mathematical benchmarks show that CCE-OPD improves avg@8 over OPD by an average of 1.60 and 1.95 percentage points when distilling Qwen3-8B into 0.6B and 1.7B students, respectively. It also achieves the strongest results on four transfer benchmarks.
| Comments: | 15 pages, 2 figures, 12 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.21195 [cs.AI] |
| (or arXiv:2610.21195v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21195 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wenxuan Luo [view email]
[v1]
Thu, 15 Oct 2026 03:55:34 UTC (255 KB)
来源：arXiv cs.AI · arxiv.org