arXiv cs.AI· Ruolin Chen, Yang Li, Jihang Wang, Zhuoya Zhao, Yuchen Liu, Jiaming Xu, Qian Zhang·· 3 天前AI 评分24
PTGPO：前缀树组策略优化
PTGPO: Prefix Tree Group Policy Optimization
AI 导读
针对长时程交互任务中稀疏终端奖励难以提供中间动作信用信号的问题，研究者提出 PTGPO（Prefix Tree Group Policy Optimization），一种无需 critic 的方法，在锚点状态组内对轨迹后缀构建前缀树，以子节点与父节点经验平均回报之差作为每条树边的信用。
正文 · 原文
Abstract:Training language agents on long-horizon interactive tasks is challenging because sparse terminal rewards provide weak credit signals for intermediate actions. Existing group-based methods either assign a shared trajectory-level advantage or estimate step-level advantages from state groups, which may conflate the current action with subsequent decisions. We propose Prefix Tree Group Policy Optimization (PTGPO), a critic-free method that constructs prefix trees over trajectory suffixes within anchor-state groups. PTGPO assigns each tree edge a credit given by the difference between its child and parent empirical mean returns, capturing the conditional return change revealed by each additional action. These prefix-tree credits provide fine-grained learning signals without requiring a critic or additional rollouts. Experiments on interactive benchmarks ALFWorld, WebShop and Sokoban show that PTGPO consistently improves over RL baselines, demonstrating its effectiveness.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14737 [cs.LG] |
| (or arXiv:2610.14737v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14737 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jihang Wang [view email]
[v1]
Sat, 10 Oct 2026 08:57:33 UTC (2,155 KB)
来源：arXiv cs.AI · arxiv.org