arXiv cs.AI· Juncheng Diao, Haibin Wang, Wei Shi, Yifan Ding, Qingbin Li, Rongxiang Weng, Xunliang Cai·· 12 小时前AI 评分38
SAPD：无需环境交互，从智能体经验中学习
SAPD: Learning from Agent Experience without Environment Interaction
AI 导读
研究者提出 Step-Advantage Policy Distillation（SAPD），一种利用相对步优势从离线轨迹中学习的方法，无需新环境观测或多轮交互。在 ALFWorld、WebShop 和 Search-based QA 上，SAPD 在不同教师配置下均提升性能，相比 SFT 缓解灾难性遗忘，每训练步比在线 OPD 至少快 4.7 倍。代码即将发布。
正文 · 原文
Abstract:Scaling the training of Large Language Model (LLM) agents typically requires a large collection of executable environments, whose construction, maintenance, and operation constrain the expansion of training experience. Existing offline trajectories offer an opportunity to reuse interaction experience, yet effectively refining the current policy without revisiting these environments remains challenging: supervised fine-tuning (SFT) can lead to catastrophic forgetting, while online reinforcement learning requires real-time interaction with the environment. To address this challenge, we propose Step-Advantage Policy Distillation (SAPD), a method for learning from offline trajectories using relative step advantages. SAPD decomposes multi-turn trajectories into individual decision contexts for on-policy distillation, allowing the student to generate responses conditioned on recorded histories, and introduces relative step advantage signals to stabilize training. The entire process requires neither new environment observations nor additional multi-turn interactions. Experiments on ALFWorld, WebShop, and Search-based QA show that SAPD improves performance across different teacher configurations, mitigates catastrophic forgetting compared with SFT, and runs at least $4.7\times$ faster per training step than online OPD. We will release our code soon.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.27847 [cs.AI] |
| (or arXiv:2610.27847v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27847 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Juncheng Diao [view email]
[v1]
Tue, 20 Oct 2026 08:30:30 UTC (938 KB)
来源：arXiv cs.AI · arxiv.org