arXiv cs.CL· Shangjian Yin, Yu Fu, Yue Dong, Zhouxing Shi·· 14 小时前AI 评分42
该奖励什么、在哪奖励？理解强化学习中可验证与开放式领域间的泛化
Where and What to Reward? Understanding Generalization Between Verifiable and Open-Ended Domains in Reinforcement Learning
AI 导读
一项控制变量研究在多个基座模型上固定优化器与训练提示词数量，仅改变提示词来源领域（数学、多领域、对话及混合）与奖励类型，以厘清 RLVR 与 RLHF 的跨域迁移。结果显示，在开放式提示词上使用奖励模型做 RLHF 不仅大幅提升对话能力，还带来推理与编码的显著增益；而 RLVR 虽提升推理，其在开放式对话上的收益远低于 RLHF。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capability of language models, since model-generated answers can be easily verified by a rule-based verifier. In contrast, reinforcement learning from human feedback (RLHF) is mainly used to optimize open-ended conversation with a learned reward model, because such responses are not easy to verify. RLVR and RLHF are usually trained separately with their own domain data and training protocols. How gains transfer between these domains is unclear, partly because comparisons change both where a model is trained (the prompts) and what is rewarded (by a verifier or a reward model). We separate the two in a controlled study of RL on multiple base models that fixes the optimizer and the number of training prompts, and varies only the source domain of the prompts (math, multi-domain, chat, or mixtures) and the reward. We find that RLHF on open-ended prompts substantially improves conversational ability and also brings large gains in reasoning and coding, whereas RLVR improves reasoning but its gains in open-ended conversation plateau far below those of RLHF, suggesting that RL on open-ended prompts with a reward model can serve as an effective approach for generalizing across domains.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.15464 [cs.LG] |
| (or arXiv:2605.15464v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15464 arXiv-issued DOI via DataCite |
Submission history
From: Shangjian Yin [view email]
[v1]
Thu, 14 May 2026 23:05:23 UTC (668 KB)
[v2]
Mon, 26 Oct 2026 17:21:42 UTC (130 KB)
来源：arXiv cs.CL · arxiv.org