arXiv cs.AI· Tiejin Chen, Xiaoou Liu, Hanyang Chen, Hua Wei·· 12 小时前AI 评分26
ST-OPSD：通过激活引导改进思考模型的自蒸馏
Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models
AI 导读
研究者提出 Steered-Teacher OPSD（ST-OPSD），通过激活引导修正 on-policy 自蒸馏中教师模型的两类问题：引用不可得参考的参考幻觉，以及抑制不确定步骤复核的倾向。
正文 · 原文
Abstract:On-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model. However, access to the reference can induce teacher behaviors that undermine independent reasoning in thinking models, leading to reference hallucinations, where the student cites an unavailable reference to justify reasoning steps, and reconsideration suppression, where the teacher discourages verification and revision at uncertain steps. We introduce Steered-Teacher OPSD (ST-OPSD) to jointly address these failure modes through activation steering. ST-OPSD extracts two directions from model-generated reasoning trajectories and applies them to the teacher's hidden states during distillation target computation, suppressing reference-invoking behavior and encouraging reconsideration with marker-family calibration. The student learns from the resulting teacher distributions without being steered. Experiments on AIME and HMMT show that ST-OPSD improves accuracy over vanilla OPSD and other interventions while restoring reconsideration and reference-hallucination rates to base-model levels.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34005 [cs.LG] |
| (or arXiv:2610.34005v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34005 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaoou Liu [view email]
[v1]
Sun, 25 Oct 2026 02:34:40 UTC (2,854 KB)
来源：arXiv cs.AI · arxiv.org