arXiv cs.AI· Yuran Wang, Zekun Wang, Bohan Zeng, Ruixu Zhang, Wenxuan Liu, Liu Yang, Yifan Dai, Yang Shi, Bozhou Li, Chengzhuo Tong, Daili Hua, Yuanxing Zhang, Wentao Zhang·· 3 天前AI 评分23
Flux-OPD：用演化上下文做在策略蒸馏
Flux-OPD: On-Policy Distillation with Evolving Contexts
AI 导读
Flux-OPD 是一种用演化上下文作为训练中监督的 OPD 范式，通过 reverse-KL 分解刻画上下文蒸馏，将上下文条件教师与无上下文教师的差异作为上下文修正注入无上下文教师，并用冲突项加权修正强度。实验显示 Flux-OPD 在 OPD 范式中取得最佳结果，表明教师监督与演化上下文结合具有潜力。
正文 · 原文
Authors:Yuran Wang, Zekun Wang, Bohan Zeng, Ruixu Zhang, Wenxuan Liu, Liu Yang, Yifan Dai, Yang Shi, Bozhou Li, Chengzhuo Tong, Daili Hua, Yuanxing Zhang, Wentao Zhang
Abstract:Large language model training in open-ended domains lacks verifiable rewards, making task preferences difficult to formalize as effective supervision. Contexts derived from task feedback can provide richer supervision, yet fixed contexts cannot adapt as the student evolves, motivating contexts that evolve with student performance. However, directly using evolving contexts as in-training supervision results in unstable distillation targets and conflicting distributions, requiring mechanisms to stabilize the target and downweight conflicts. In this paper, we characterize context distillation through a reverse-KL decomposition, showing that the student is distilled toward the geometric mean of context-conditioned teachers, while the objective contains a conflict term measuring disagreement among these teachers. Motivated by this characterization, we propose Flux-OPD, an OPD paradigm that uses evolving contexts as in-training supervision to incorporate contextual signals derived from task feedback. Flux-OPD treats differences between context-conditioned and context-free teachers as contextual difference signals, injects them as contextual corrections into the context-free teacher, and weights the correction strength using the conflict term. Experiments show that Flux-OPD achieves the best results among OPD paradigms, highlighting the potential of combining teacher supervision with evolving contexts as in-training supervision.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.28022 [cs.LG] |
| (or arXiv:2607.28022v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.28022 arXiv-issued DOI via DataCite |
Submission history
From: Yuran Wang [view email]
[v1]
Thu, 30 Jul 2026 11:11:32 UTC (2,758 KB)
[v2]
Mon, 12 Oct 2026 06:47:25 UTC (2,769 KB)
来源：arXiv cs.AI · arxiv.org