arXiv cs.AI· Zhengpeng Xie, Li Lyna Zhang, Zeke Xie, Mao Yang·· 13 小时前AI 评分31
TOP-D：将在线策略蒸馏转化为稳定训练范式的信任区域策略蒸馏
Trust Region Policy Distillation
AI 导读
研究者提出 Trust Region Policy Distillation（TOP-D），通过动态构建近端教师模型，将高方差的 On-Policy Distillation（OPD）转化为稳定训练范式。理论分析表明 TOP-D 可控制梯度方差，并给出全局收敛分析与单调改进界；在数学推理任务上，其训练稳定性、样本效率和最终性能均显著提升，且不引入额外计算开销。
正文 · 原文
Abstract:Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable, high-variance On-Policy Distillation (OPD) into a stable training paradigm by dynamically constructing a proximal teacher. Theoretically, we establish a rigorous framework demonstrating that TOP-D inherently controls gradient variance. By providing a formal global convergence analysis alongside a monotonic improvement bound, we mathematically formalize the reliability and stability of the overall training dynamics. Empirically, TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. More importantly, TOP-D introduces zero additional computational overhead, positioning itself as a promising alternative to the well-established OPD paradigm.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.04751 [cs.LG] |
| (or arXiv:2607.04751v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.04751 arXiv-issued DOI via DataCite |
Submission history
From: Zhengpeng Xie [view email]
[v1]
Mon, 6 Jul 2026 07:43:36 UTC (129 KB)
[v2]
Mon, 26 Oct 2026 23:22:57 UTC (141 KB)
来源：arXiv cs.AI · arxiv.org