arXiv cs.CL· Mingchen Li, Hansi Zeng, Zhuo Qian, Jiatan Huang, Sunjae Kwon, Hamed Zamani, Hong Yu·· 9 小时前AI 评分31
RICE-PO:将检索交互转化为推理智能体的信用信号
RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents
AI 导读
RICE-PO 是一个无需 critic 的策略优化框架,把检索交互转化为局部学习信号,用于训练迭代检索的推理智能体。它选取高不确定性的可执行动作作为锚点,用检索指标评估局部反事实分支,仅在推理对动作影响强且未来残差效应稳定时向隐式推理步骤传播信用。在 BRIGHT 和 BEIR 上,RICE-PO 在相同检索器设置下持续优于基于提示词的智能体和基于分组的 RL 基线。
正文 · 原文
Abstract:Retrieval is increasingly moving from one-shot matching toward interactive reasoning, where language agents iteratively inspect evidence, reformulate queries, and search again. Training such agents raises a credit-assignment challenge: executable actions such as queries or summaries can be directly evaluated by the retriever, while latent reasoning steps are not directly observable and only affect future executable actions. This asymmetry makes outcome-level reward assignment unreliable, as the same final reward may credit reasoning steps that did not actually shape retrieval success. We propose RICE-PO, a critic-free policy optimization framework that converts retrieval interactions into localized learning signals. RICE-PO selects high-uncertainty executable actions as anchors, evaluates local counterfactual branches using retrieval metrics, and propagates credit to latent reasoning steps only when reasoning-to-action influence is strong and future residual effects are stable. On BRIGHT and BEIR, RICE-PO consistently outperforms prompt-based agents and group-based RL baselines under the same retriever setting. These results show that the structure of agent-environment interaction itself can provide useful supervision for training reasoning-based retrieval agents.
| Comments: | NIPS 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.26352 [cs.CL] |
| (or arXiv:2605.26352v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.26352 arXiv-issued DOI via DataCite |
Submission history
From: Mingchen Li [view email]
[v1]
Mon, 25 May 2026 21:56:29 UTC (215 KB)
[v2]
Sat, 1 Aug 2026 16:06:15 UTC (215 KB)
[v3]
Thu, 6 Aug 2026 01:35:38 UTC (215 KB)
[v4]
Tue, 17 Nov 2026 18:16:21 UTC (194 KB)
来源:arXiv cs.CL · arxiv.org