跳到正文
arXiv cs.AI· Siyuan Du, Yuxin Liu, Meng Tang, Haoran Li, Yechao Bai, Feiyang Pan, Jia He·· 10 小时前AI 评分31

从后往前:面向 LLM 智能体的逆向到正向课程与优势分解

Depth Before Breadth: Reverse-to-Forward Curriculum with Advantage Decomposition for LLM Agent

AI 导读

研究提出“策略诱导能力区域”视角,用配对优势与逆向到正向渐进课程解决 LLM 智能体稀疏终端奖励下的信用分配与探索难题。在 ALFWorld 和 WebShop 上,该方法在 Qwen2.5-1.5B/7B-Instruct 两个规模均优于竞争性智能体 RL 方法,成功率分别达 90.1%/96.1% 和 80.2%/82.6%。

正文 · 原文

View PDF HTML (experimental)

Abstract:Reinforcement learning (RL) has become a common approach for training large language model (LLM) agents on long-horizon tasks. However, learning from sparse terminal rewards makes credit assignment and exploration difficult. We approach these challenges through a policy-induced capability region, the set of states from which the current policy can reliably complete a task. Near its boundary, an agent may need only a few actions to reach a more solvable state, even when the goal remains distant. This reduces long-horizon learning to two coupled subproblems of reaching that state and completing the task from there. We address them with paired advantages and a progressive reverse-to-forward curriculum. The advantages provide separate signals for reaching more solvable states and completing the task from them, while the curriculum expands the capability region backward along successful reference trajectories and then into off-reference states through on-policy exploration. On ALFWorld and WebShop, our method outperforms competitive agentic RL methods at both model scales, achieving success rates of 90.1%/96.1% and 80.2%/82.6% with Qwen2.5-1.5B/7B-Instruct, respectively.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24697 [cs.AI]
  (or arXiv:2611.24697v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.24697

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Meng Tang [view email]
[v1] Wed, 18 Nov 2026 16:25:14 UTC (1,226 KB)

来源:arXiv cs.AI · arxiv.org