arXiv cs.AI· Yingdong Shi, Zhihong Li, Changming Li, Jingyi Yu, Quan Li, Kan Ren·· 3 天前AI 评分26
OPMD：超越输出监督的在线策略机制蒸馏
OPMD: On-Policy Mechanism Distillation Beyond Output Supervision
AI 导读
研究者提出在线策略机制蒸馏（OPMD），通过先构建与学生兼容的教师模型，将教师中间表示转化为学生下游计算可用的状态，再在在线策略轨迹中内化这些状态，从而在输出监督之外实现机制级监督。在三种教师-学生设定中，OPMD 的平均准确率均高于所评估的蒸馏基线，并可泛化到多教师蒸馏和跨家族监督微调。分析显示，机制监督能带来更有效的推理，并显著改变学生的中间计算与解题策略。
正文 · 原文
Abstract:On-policy distillation (OPD) improves language-model by supervising a student on prefixes sampled from its own policy, but its supervision remains confined to the output space. Consequently, OPD transfers what a teacher predicts without supervising the intermediate computation that produces those predictions. Simply distilling intermediate representations like representation matching is insufficient since teacher and student representations may differ geometrically, and even well-aligned hidden states need not be usable by the student's subsequent computation. We introduce On-Policy Mechanism Distillation (OPMD), which transfers teacher-derived intermediate computation while explicitly accounting for its compatibility with the student. OPMD first constructs a student-compatible teacher that transforms teacher representations into intermediate states usable by the student's downstream computation, then it internalizes these states from on-policy trajectories. OPMD achieves the highest average accuracy among the evaluated distillation baselines in all three teacher--student settings, and generalizes to multi-teacher distillation and cross-family supervised fine-tuning. Further analyses show that mechanism supervision yields more effective reasoning and meaningfully alters the student's intermediate computation and solution strategies, supporting that it transfers information beyond output supervision alone. Together, these results establish mechanism-level supervision as an effective way to move on-policy distillation beyond output supervision, enabling students to learn inner mechanisms they can use and internalize.
| Comments: | 21 pages, 6 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14022 [cs.LG] |
| (or arXiv:2610.14022v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14022 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingdong Shi [view email]
[v1]
Tue, 29 Sep 2026 14:07:18 UTC (615 KB)
来源：arXiv cs.AI · arxiv.org