arXiv cs.AI· Hao Li, Zhengkun Zhang, Gangqiang Hu, Zhen Zhang, Yude Gao, Dai Dai, Jing Liu·· 6 天前AI 评分29
RLVR²：基于可验证评分标准排序的强化学习
RLVR$^{2}$: Reinforcement Learning with Verifiable Rubric-based Ranking
AI 导读
针对多维度评分标准下 RLVR 的奖励聚合问题，研究者提出 RLVR²，将各评分标准的分数转为组内序数结果，从比较矩阵中恢复潜在效用并合并为单一训练信号，避免校准异质评分尺度。在 3 种模型规模和 16 个 benchmark 上，RLVR² 均优于代表性评分标准基线，在多数 benchmark 上取得最佳整体表现。该方法还支持在不扩展评分标准的情况下引入与排序相关但非训练目标的辅助属性。
正文 · 原文
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) is expanding from tasks with well-defined correctness signals, such as mathematics and code, toward multifaceted quality requirements specified by multi-dimensional rubrics. Since policy optimization consumes one scalar per rollout, rubric-based pipelines must map multiple criterion scores into a scalar reward. This aggregation is often treated as score scaling, but it implicitly determines how quality dimensions trade off during training. The prevailing practice, normalizing each criterion and taking a linear combination, assumes that cardinal score differences are comparable across criteria and that gains on one criterion compensate for failures on another; both assumptions are unreliable when criteria are semantically heterogeneous. We propose Reinforcement Learning with Verifiable Rubric-based Ranking (RLVR$^2$), a verifiable ranking paradigm for rubric-based RLVR. For each criterion, RLVR$^2$ converts rubric scores into criterion-specific within-group ordinal outcomes, recovers a latent utility from the resulting comparison matrix, and merges these utilities into one training signal. By retaining only within-group ordering and discarding raw score magnitudes, RLVR$^2$ avoids calibrating heterogeneous rubric scales. It further supports objective-preserving attribute adjustment: auxiliary attributes that correlate with observed rankings but are not training objectives can enter the estimation without expanding the rubric or rewarding them directly. Across three model scales and 16 benchmarks, RLVR$^2$ consistently outperforms representative rubric-based baselines, achieving the best overall performance on most benchmarks at every scale. Analysis shows it controls systematic effects tied to reasoning efficiency and response formatting while preserving the quality objective.
| Comments: | Preprint |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.23457 [cs.LG] |
| (or arXiv:2609.23457v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.23457 arXiv-issued DOI via DataCite |
Submission history
From: Hao Li [view email]
[v1]
Sun, 20 Sep 2026 08:34:28 UTC (2,132 KB)
[v2]
Fri, 9 Oct 2026 10:31:18 UTC (5,400 KB)
来源：arXiv cs.AI · arxiv.org