跳到正文
arXiv cs.AI· Wen Wang, Jindong Wang, Jiahua Bao, Tu Yongsiqi, Yihao Liu, Junwei He, Wenkui Fan·· 10 小时前AI 评分31

基于证据的信用分配:提升搜索增强语言模型的强化学习效果

Evidence-Based Credit Assignment for Search-Augmented Language Models

AI 导读

研究提出 Evidence-based Credit Assignment(ECA),用仅含问题与已检索文档的辅助信号衡量检索证据对参考答案似然的提升,替代对整段对话前缀打分。

正文 · 原文

View PDF HTML (experimental)

Abstract:Reinforcement learning from terminal answer rewards enables search-augmented language models to interleave reasoning and retrieval without step-level annotations, yet outcome supervision assigns identical credit to every search in a trajectory regardless of whether it retrieved useful information. This limitation is especially severe when all sampled rollouts for a question fail, collapsing group-centered outcome advantages to zero even when trajectories gather markedly different evidence. We present Evidence-based Credit Assignment (ECA), which augments terminal outcome rewards with a training-only auxiliary signal measuring how retrieved documents improve the likelihood of a reference answer. Rather than scoring the full dialogue prefix, ECA queries a frozen snapshot of the current policy under an evidence-only prompt containing solely the question and the deduplicated documents observed so far, isolating the utility of retrieved passages from surface variations in generated thoughts or search queries. Within this framework, we compare two return formulations, ECA-Step (which accumulates step-clipped potential increments) and ECA-Horizon (which preserves the telescoping horizon gain in potential), paired with cross-fold group normalization, novelty gating, and search-local token assignment. Across seven Search QA benchmarks, ECA-Horizon improves strict macro exact match over an outcome-only GRPO baseline by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct, respectively, outperforming both RLOO and GiGPO. Zero-shot evaluation on MultiHop-RAG further demonstrates strong transfer to multi-hop inference queries.
Comments: 16 pages, 4 figures. Includes supplementary material
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23824 [cs.AI]
  (or arXiv:2611.23824v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.23824

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Wen Wang [view email]
[v1] Wed, 18 Nov 2026 04:24:51 UTC (1,294 KB)

来源:arXiv cs.AI · arxiv.org