arXiv cs.AI· Yibin Huang, Conghui Zhu·· 12 小时前AI 评分22
Distill Until Divergence：面向智能体强化学习的错误截断事后蒸馏方法
Distill Until Divergence: Error-Truncated Hindsight Distillation for Agentic RL
AI 导读
针对基于结果的 GRPO 缺乏中间决策细粒度指导、技能条件监督在错误动作后与当前状态失配的问题，研究者提出 Distill Until Divergence 方法，按轨迹结果与错误边界路由正向与负向两类事后技能：正向技能强化成功轨迹及出错前决策，负向技能针对决定性错误提供纠正，错误后的后缀仅由 GRPO 优化。实验显示该方法取得有竞争力的任务表现，并支持有效的细粒度信用分配。
正文 · 原文
Abstract:Long-horizon interaction requires agents to make multiple decisions in a changing environment. Outcome-based GRPO lacks fine-grained guidance for intermediate decisions. Recent methods distill completed trajectories into reusable skills and use hindsight-conditioned OPD to provide dense token-level credit. However, skill-induced probability shifts do not necessarily represent reward-aligned credit; especially after an erroneous action changes subsequent interaction states, skill-conditioned supervision may become mismatched with the current state. To address this mismatch, we propose \methodname{}, which routes two complementary hindsight skills according to trajectory outcomes and error boundaries: Positive skills capture successful workflows to reinforce successful trajectories and pre-error decisions, while Negative skills capture failure-specific corrections to target decisive errors. The post-error suffix is optimized exclusively by GRPO, retaining outcome-based learning without extending potentially mismatched hindsight supervision. Extensive experiments demonstrate competitive task performance and support effective fine-grained credit allocation.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.23048 [cs.AI] |
| (or arXiv:2610.23048v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23048 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yibin Huang [view email]
[v1]
Fri, 16 Oct 2026 07:28:05 UTC (549 KB)
来源：arXiv cs.AI · arxiv.org