arXiv cs.CL· Linfei Ji (Huazhong University of Science and Technology), Jiayun Li (Tsinghua University), Haozhao Wang (Huazhong University of Science and Technology), Siyu Jiang (Huazhong University of Science and Technology), Zehao Wang (Harbin Institute of Technology), Longtao Xiao (Huazhong University of Science and Technology), Yu-Jie Yuan (Huawei Technologies Ltd), Ruixuan Li (Huazhong University of Science and Technology)·· 13 小时前AI 评分24
TeachBack：用双向偏好优化实现 On-Policy 自蒸馏的教学相长
TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation
AI 导读
华中科技大学等团队提出 TeachBack，一个面向 On-Policy 自蒸馏（OPSD）的双向偏好学习框架，让模型不仅向 Teacher 学习，也学习 Teacher 该如何教。
正文 · 原文
Authors:Linfei Ji (1), Jiayun Li (2), Haozhao Wang (1), Siyu Jiang (1), Zehao Wang (3), Longtao Xiao (1), Yu-Jie Yuan (4), Ruixuan Li (1) ((1) Huazhong University of Science and Technology, (2) Tsinghua University, (3) Harbin Institute of Technology, (4) Huawei Technologies Ltd.)
Abstract:On-policy self-distillation (OPSD) enables a language model to learn from a Teacher instantiated from the same model but given access to privileged information, such as a reference solution. We find that access to sufficient privileged information allows the Teacher to generate highly accurate trajectories; however, accuracy alone does not guarantee that these trajectories are suitable for Student learning. Direct access to a complete solution may encourage shortened, reference-conditioned derivations that omit the exploration, verification, and self-correction required to solve problems without privileged information. We introduce TeachBack, a bidirectional preference-learning framework for on-policy self-distillation that learns not only from the Teacher but also how the Teacher should teach. TeachBack shapes the Teacher's generation behavior by controlling the length of the reference-reasoning suffix injected into the prompt and by varying the prompt style. While maintaining comparable Teacher accuracy, it encourages the Teacher to explore and re-solve problems, thereby reducing its reliance on the reference solution. At each rollout refresh, the shared policy generates both a privileged Teacher trajectory and a problem-only Student trajectory. A pairwise preference objective uses privileged Teacher trajectories to improve Student learning, while the reversed preference uses the Student's behavior to shape future Teacher supervision under the privileged condition. Experiments on mathematical and logical reasoning tasks show that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.
| Comments: | 21 pages, 6 figures, including appendices |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.29167 [cs.AI] |
| (or arXiv:2610.29167v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29167 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Linfei Ji [view email]
[v1]
Wed, 21 Oct 2026 03:59:23 UTC (2,708 KB)
来源：arXiv cs.CL · arxiv.org