arXiv cs.AI· Lianfa Li·· 2 天前AI 评分24
面向 LLM 算术推理的 Bellman 一致性轨迹 RL
Bellman-Consistent Trajectory RL for LLM Arithmetic Reasoning
AI 导读
针对 LLM 算术推理中终端奖励稀疏的问题，研究者提出 Bellman 一致性轨迹 RL 方法 TBRM，将稀疏奖励转化为密集的 token 级信号，并结合 RLOO 分组策略降低 Bellman 残差的梯度方差、用事后目标重标注加速中间学习。
正文 · 原文
Abstract:To address sparse terminal rewards in LLM arithmetic reasoning, we propose a Bellman-consistent trajectory RL method. Our framework introduces Trajectory Bellman Residual Minimization (TBRM) to transform sparse rewards into dense token-level signals. We integrate a Reinforce Leave-One-Out (RLOO) group strategy to reduce gradient variance of Bellman residuals and use hindsight goal relabeling to accelerate intermediate learning. Evaluated on the Countdown task, our method alleviates optimization bottlenecks and stabilizes gradient variance significantly better than GRPO. The TBRM-RLOO framework boosts Pass@1 to 0.55--0.64 and achieves a state-of-the-art Pass@16 accuracy of 76.0--84.0\%, outperforming the SFT, IPO/DPO, and RLOO baselines. Our approach demonstrates exceptional single-sample efficiency and successfully circumvents standard performance plateaus via test-time scaling.
| Comments: | Grade A as a project report |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.21060 [cs.LG] |
| (or arXiv:2610.21060v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21060 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lianfa Li [view email]
[v1]
Thu, 15 Oct 2026 01:05:17 UTC (3,233 KB)
来源：arXiv cs.AI · arxiv.org