arXiv cs.AI· Haowei Guo, Baolong Bi, Ruicheng Zhang, Cong Li, Bingqian Sun, Zhonghai Wu·· 6 天前AI 评分24
教师模型何时该更新？自同策略蒸馏中的时间耦合与稳定性
When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation
AI 导读
研究者在 Qwen3-8B 上通过受控调度扫描发现，教师模型更新之间的完全冻结期（隔离期）才是稳定学习的关键，而非教师年龄。团队提出时间 KL 结构、刷新冲击与长度尾部风险诊断框架，揭示短视评估下不可见的“状态无视崩溃”，并提出 CGTR 方法，将刷新门控于奖励提升与长度尾部安全的联合证据上。
正文 · 原文
Abstract:Self on-policy distillation trains a student policy against a teacher derived from its own parameter history, yet the teacher's update schedule -- which governs the \emph{temporal coupling} between teacher and student -- has not been systematically studied as a stability variable. Through a controlled schedule sweep on Qwen3-8B, we establish that \emph{isolation periods}, defined as complete teacher freezing between updates, are the key structural property enabling stable learning, not teacher age. To characterize these underlying training dynamics, we introduce a diagnostic framework of temporal KL structure, refresh shock, and length-tail risk. This framework further uncovers \emph{state-oblivious collapse}: optimal short-horizon fixed schedules catastrophically fail under long-horizon training because a clock-driven refresh can copy a transiently drifting student into the teacher in a single, irreversible step. This failure mode is invisible under short-horizon evaluation and mechanistically distinct from EMA's chronic contamination. To address this, we propose \emph{Consolidation-Gated Teacher Refresh} (CGTR), which preserves isolation periods while gating each refresh on joint evidence of reward improvement and length-tail safety, ensuring every teacher movement responds to genuine student consolidation rather than a clock signal. With a single shared parameter set and no per-dataset retuning, CGTR achieves \textbf{zero collapse} and the best final score on all four tasks (Chemistry, Biology, Physics, ToolUse), self-regulating its refresh frequency to each task's learning dynamics.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.03532 [cs.LG] |
| (or arXiv:2606.03532v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.03532 arXiv-issued DOI via DataCite |
Submission history
From: Haowei Guo [view email]
[v1]
Tue, 2 Jun 2026 11:54:39 UTC (1,233 KB)
[v2]
Fri, 9 Oct 2026 13:22:21 UTC (1,103 KB)
来源：arXiv cs.AI · arxiv.org