arXiv cs.CL· Minchan Kim, Sungho Bae, Youngjin Kim, Hyunwoo Park·· 3 天前AI 评分22
GRPO 长程 LLM 智能体训练该从哪里起步、如何推进：课程学习策略分析
Where to Start, How to Move: Analyzing Curricula under GRPO for Long-Horizon LLM Agents
AI 导读
研究分析了 Guide Assisted GRPO 在长程 LLM 智能体任务中的课程设计，考察初始位置、交接单元和移动规则对无辅助任务完成率的影响。在 BFCL 上跨三种模型规模，固定轮级阈值下从接近完成处起步更有效；更细粒度的交接本身帮助不稳定，但基于结果信息的工具调用选择在已训练和未训练任务类别上均优于 GRPO。引导式起步在 AppWorld 两个评测划分上也提升了成功率。
正文 · 原文
Abstract:Group Relative Policy Optimization (GRPO) provides no reward-relative learning signal when all rollouts receive identical rewards. In long-horizon tasks that require multiple interactions, rewards are given only upon successful task completion, making all-fail groups common early in training and limiting opportunities to practice later steps. Guide Assisted GRPO replays a successful reference prefix before the learner continues. We study how initial positions, handoff units, and movement rules affect unassisted task completion. Across three model sizes on BFCL, starting near completion is more effective under a fixed turn-level threshold rule. Finer handoffs alone do not consistently help, but outcome-informed tool-call selection outperforms GRPO on trained and untrained task categories. Guided starts also improve AppWorld success on both evaluation splits. These findings highlight where learning starts and how starting positions move as key curriculum choices.
| Comments: | 7 pages, 4 figures, 4 tables. Minchan Kim and Sungho Bae contributed equally |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19349 [cs.LG] |
| (or arXiv:2610.19349v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19349 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Minchan Kim [view email]
[v1]
Wed, 14 Oct 2026 00:42:53 UTC (205 KB)
来源：arXiv cs.CL · arxiv.org