arXiv cs.AI· Shufan Shen, Zhongni Hou, Chao Bi, Yufei Zhang, Guojun Yin, Shuhui Wang·· 14 小时前AI 评分31
Teacher-Evolution Self-Distillation（TESD）：让自教师随训练进化的在线自蒸馏方法
On-Policy Self-Distillation with Teacher Evolution
AI 导读
针对 On-Policy Self-Distillation（OPSD）训练中性能难以持续提升的问题，研究者提出 Teacher-Evolution Self-Distillation（TESD），让自教师基于任务正确性与上下文独立性获得环境反馈并持续优化，同时按教师指导可靠性对不可靠样本降权。多数据集、多模型规模实验显示 TESD 稳定优于代表性 token 级与序列级方法。
正文 · 原文
Abstract:On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity in sequence-level post-training paradigms. However, OPSD often fails to achieve continuous performance gains during training. In this paper, we find that the model's limited self-teaching capability underlies this phenomenon by generating unreliable guidance on certain samples. To address this problem, we propose Teacher-Evolution Self-Distillation (TESD), which enables the self-teacher to generate rollouts and receive environmental feedback based on task correctness and context independence. First, using these rewards as training objectives, we perform teacher-oriented optimization to continuously enhance the model's self-teaching capability during distillation. Second, by estimating teacher guidance reliability from these rewards, we apply adaptive student-oriented optimization to down-weight samples with unreliable guidance. Extensive experiments across multiple datasets and model scales show that TESD consistently outperforms representative token-level and sequence-level methods, highlighting the necessity of improving supervision quality in OPSD.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38637 [cs.LG] |
| (or arXiv:2610.38637v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38637 arXiv-issued DOI via DataCite |
Submission history
From: Shufan Shen [view email]
[v1]
Wed, 28 Oct 2026 10:51:52 UTC (2,983 KB)
来源：arXiv cs.AI · arxiv.org