跳到正文
arXiv cs.AI· Shuai Zhao, Yunqiu Xu, Linchao Zhu, Ling Chen, Yi Yang·· 10 小时前AI 评分34

RL2R:用基于参考的奖励统一语言模型推理与对齐的强化学习

Reinforcement Learning with Reference-Based Rewards for Language Model Reasoning and Alignment

AI 导读

研究者提出 RL2R,用 BERTScore 等轻量指标衡量生成结果与已有或蒸馏参考答案的语义相似度作为奖励,统一替代推理与对齐中的奖励设计。在数学推理上,答案级与步骤级相似度可提供低成本过程监督,优于仅验证最终答案;对齐任务上,无 critic 的 REINFORCE 式实现可覆盖通用偏好、安全与置信度任务,效果与基于 RM 的方法相当。代码已公开。

正文 · 原文

View PDF HTML (experimental)

Abstract:Reinforcement learning (RL) powers two major lines of large language model (LLM) post-training: reasoning, which elicits deliberate, verifiable problem solving, and alignment, which steers models toward helpful, harmless, and honest behavior. Both hinge on reward design. Reasoning relies on rule-based verification of final answers, whose binary feedback is blind to intermediate derivations, whereas process reward models (PRMs) require step-level annotations too costly to scale. Alignment depends on reward models (RMs) trained on binary preference pairs, making both data collection and model training resource-intensive. We propose RL2R, reinforcement learning with reference-based rewards, as a unified and inexpensive alternative. Using pre-existing or distilled references, RL2R measures semantic similarity with lightweight metrics such as BERTScore between a sampled generation and a reference answer or solution. Similarity preserves ordinal quality signals that binary feedback discards: responses closer to the reference earn higher rewards, and sound derivations receive credit even when the final conclusion is wrong. For mathematical reasoning, augmenting rule-based verification with answer- and step-level similarity provides low-cost process supervision and outperforms answer-only verification. For alignment, a critic-free REINFORCE-style instantiation covers general preference, safety, and confidence tasks on par with RM-based methods. Reference-based rewards thus offer a cost-effective, versatile reward source spanning reasoning and alignment.
Comments: The code is at this https URL. This version adds a reasoning part with reference-based rewards compared to the previous version
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2504.09895 [cs.CL]
  (or arXiv:2504.09895v4 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2504.09895

arXiv-issued DOI via DataCite

Submission history

From: Shuai Zhao [view email]
[v1] Mon, 14 Apr 2025 05:43:21 UTC (414 KB)
[v2] Sun, 22 Jun 2025 13:02:25 UTC (337 KB)
[v3] Mon, 13 Oct 2025 05:00:17 UTC (466 KB)
[v4] Wed, 18 Nov 2026 11:06:19 UTC (2,156 KB)

来源:arXiv cs.AI · arxiv.org