arXiv cs.CL· Sohyun Son, Yerim Hong, Jaeyun Park, Mujeen Sung·· 13 小时前AI 评分31
RoAR：用 rollout 一致性改进 GRPO 推理训练
Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization
AI 导读
研究者提出 RoAR（Rollout-Agreement Reward），在 GRPO 的奖励中加入推理层面的一致性信号，且无需人工撰写理由、额外采样或单独训练的过程评估器。
正文 · 原文
Abstract:Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficiency. Despite these gains, the extent to which differences among correct-answer rollouts reflect useful alternative reasoning or unresolved errors remains poorly understood. We investigate this question through a blind paired analysis of a model trained with Group Relative Policy Optimization (GRPO), comparing the highest- and lowest-agreement correct-answer traces for the same problem. Across 126 pairs, low-agreement traces have more spurious reasoning than high-agreement traces (35% vs. 22%), including internal inconsistencies and misuse of problem conditions. Based on these findings, we propose RoAR (Rollout-Agreement Reward), which adds reasoning-level agreement to GRPO's rewards for verified-correct rollouts. RoAR computes this signal from existing on-policy rollout groups, requiring no human-written rationales, additional sampling, or separately trained process evaluator. On five mathematical reasoning benchmarks, the 1.5B GRPO-RoAR model reaches 55.5% mean pass@1 under zero-shot evaluation using 7K training examples, exceeding GRPO and GRPO-CARE by 6.0 and 5.5 percentage points. A separate blind audit of 100 problems answered correctly by all four 1.5B models finds spurious rates of 18% for GRPO-RoAR, 32% for GRPO, 27% for GRPO-CARE, and 37% for DRA-GRPO. Our findings identify reasoning validity as an important consideration when using rollout differences to guide learning. Our code is available at this https URL.
| Comments: | 21 pages, 7 figures, 16 tables |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34951 [cs.LG] |
| (or arXiv:2610.34951v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34951 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sohyun Son [view email]
[v1]
Mon, 26 Oct 2026 07:19:26 UTC (373 KB)
来源：arXiv cs.CL · arxiv.org