arXiv cs.AI· Xianwei Chen, Shimin Zhang, Jibin Wu·· 3 天前AI 评分22
f-OPD：用新鲜度感知控制稳定长程在线策略蒸馏
$\boldsymbol{f}$-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
AI 导读
针对 LLM 在线策略蒸馏（OPD）中异步执行偏离理想 on-policy 目标的问题，研究者提出 f-OPD 框架，将目标偏差分解为 rollout drift 与 supervision drift，并引入样本级新鲜度分数衡量缓冲样本可靠性，据此自适应调节陈旧样本影响、约束策略漂移。
正文 · 原文
Abstract:Scaling on-policy distillation (OPD) for large language models (LLMs) confronts a fundamental tension: asynchronous execution is necessary for system efficiency, but structurally deviates from the ideal on-policy objective. To address this challenge, we theoretically decompose the objective discrepancy into rollout drift and supervision drift, capturing staleness in student rollout and teacher context, respectively. Building on this, we introduce a sample-level freshness score that quantifies the reliability of a buffered sample with respect to the on-policy objective. Guided by this signal, we further propose f-OPD, a novel framework that adaptively regulates stale-sample influence and constrains policy drift accumulated under asynchronous training. Across reasoning, tool-use, and coding-agent tasks of increasing interaction horizon, f-OPD consistently achieves task performance comparable to synchronous optimization while largely retaining the throughput advantages of asynchronous execution. Our results establish the first recipe for achieving a performance-efficiency trade-off in OPD, paving the way for long-horizon agentic post-training at scale.
| Comments: | This work has been accepted at NeurIPS 2026. This is the final camera-ready version |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.17862 [cs.LG] |
| (or arXiv:2605.17862v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.17862 arXiv-issued DOI via DataCite |
Submission history
From: Xianwei Chen [view email]
[v1]
Mon, 18 May 2026 05:14:18 UTC (1,069 KB)
[v2]
Sun, 11 Oct 2026 15:23:55 UTC (959 KB)
来源：arXiv cs.AI · arxiv.org