arXiv cs.AI· Mengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou·· 11 小时前AI 评分34
AttnSR：用注意力步骤奖励改进长程智能体强化学习
AttnSR: Attention Step Reward for Long-Horizon Agentic RL
AI 导读
AttnSR 是一种步骤级奖励增强方法，直接从训练回放中提取策略注意力来分配各步骤奖励，无需单独训练奖励模型或额外 rollout。在 Qwen2.5-7B-Instruct 的对照实验中，AttnSR 在 ALFWorld 所有评估 checkpoint 上均优于 PPO 基线，最终留出集成功率从 91.2% 提升至 97.1%。
正文 · 原文
Abstract:Long-horizon LLM agents make multi-step decisions, yet outcome-reward-only reinforcement learning provides little direct supervision for individual steps. We introduce AttnSR, a step-level reward augmentation method that derives reward allocations from policy attention extracted during training replay, without a separately trained reward model or additional rollouts. AttnSR traces backward through the policy model attention graph to estimate the relevance of earlier steps to the final response. The terminal step, measured relative to a running baseline, determines whether these rewards reinforce or penalize the selected steps. In a controlled Qwen2.5-7B-Instruct study, AttnSR consistently outperforms the PPO baseline across all evaluated checkpoints on ALFWorld, raising final held-out success from 91.2% to 97.1%. The ablation experiment shows that the benefit depends on how intermediate rewards are assigned, rather than simple accumulation of step rewards over time. AttnSR also improves ScienceWorld at every checkpoint while cutting environment interactions by 29.9%. These results support policy attention as a practical prior for step-level reward allocation in long-horizon agentic RL. Code is available at this https URL .
| Comments: | 20 pages, 4 figures, including appendices. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07932 [cs.LG] |
| (or arXiv:2611.07932v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07932 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mengsong Wu [view email]
[v1]
Thu, 5 Nov 2026 12:37:19 UTC (371 KB)
来源：arXiv cs.AI · arxiv.org