arXiv cs.AI· Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah·· 11 小时前AI 评分42
Rubric Curriculum RL：利用生成-验证差距提升非可验证领域强化学习
Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains
AI 导读
研究者提出 Rubric Curriculum RL（RcRL），无需新数据、人工标注或更强教师模型，通过成对偏好比较与评分标准课程相结合，在开放式领域获得更具判别力的非平稳训练信号。
正文 · 原文
Authors:Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah
Abstract:Reinforcement learning with verifiable rewards (RLVR) has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. The generation-verification gap, where judging good outputs is easier than producing them, offers a potential path to improvement. However, rubric-based judges are often imprecise and stationary, so policies can overfit or reward hack them. We introduce Rubric Curriculum RL (RcRL), which exploits this gap without new data, human annotations, or stronger teacher models. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring and a non-stationary objective. Whereas baselines plateau within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL achieves a 70.5% human win rate against the base model, with gains across benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all training baselines, including instance-specific rubrics.
| Comments: | Accepted at ICML 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06244 [cs.LG] |
| (or arXiv:2611.06244v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06244 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sumeet Motwani [view email]
[v1]
Wed, 4 Nov 2026 09:51:06 UTC (472 KB)
来源：arXiv cs.AI · arxiv.org