arXiv cs.AI· Yingjian Chen, Lang Zhou, Zixin Zhao, Pingfan Li, Weishi Zheng, Zhilin Zhao·· 13 小时前AI 评分31
ConCal：用延续共享校准动作价值，提升长程智能体强化学习
Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning
AI 导读
针对长程智能体强化学习中信用分配困难的问题，研究者提出 ConCal（Continuation Calibration），一种递归延续价值估计器，利用 rollout 批次中已有的延续证据校准动作价值，无需额外采样。
正文 · 原文
Abstract:Reinforcement learning for long-horizon agents relies on feedback from sequences of interactions, making credit assignment to individual actions difficult. Existing methods such as GiGPO compare actions at shared states to compute relative advantages, but estimate each action's value from a single sampled continuation. The resulting local credit remains sensitive to decisions made later in the trajectory. To address this problem, we propose ConCal (Continuation Calibration), a recursive continuation-value estimator that incorporates continuation evidence already present in a rollout batch. ConCal matches visits from the same task by their incoming state-action-state transitions, combines shared continuation evidence with each visit's own transition estimate, and propagates calibrated values backward to refine earlier action-value estimates. This reduces dependence on individual sampled continuations and mitigates local credit distortions without additional rollouts. Held-out return diagnostics show lower error in within-group value differences and higher agreement with reference action orderings than estimates based on individual continuations. With Qwen2.5-1.5B-Instruct, ConCal improves success rates over GiGPO by 3.39 and 5.86 percentage points on ALFWorld and WebShop, respectively. Our code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37195 [cs.LG] |
| (or arXiv:2610.37195v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37195 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingjian Chen [view email]
[v1]
Tue, 27 Oct 2026 13:16:08 UTC (2,991 KB)
来源：arXiv cs.AI · arxiv.org