arXiv cs.AI· Yeonbum Kim, Hyun Kyu Lee, Sung Whan Yoon·· 13 小时前AI 评分31
Flat-RLVR：面向 LLM 推理能力的锐度感知可验证奖励强化学习
Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs
AI 导读
Flat-RLVR 通过锐度感知最小化在 LLM 后训练中寻找平坦奖励区域，以缓解 RLVR（尤其是 GRPO）带来的分布锐化问题。该方法在理论上将平坦奖励 RLVR 与分布锐化减弱相关联，实验显示在保持单次尝试准确率的同时提升了样本多样性，并与 GRPO 变体结合以权衡采样效率与推理能力。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR), particularly through group-relative policy optimization (GRPO), has emerged as a central paradigm for the post-training of large language models (LLMs). While RLVR is highly effective at enhancing the reasoning capabilities of LLMs, it commonly induces \textit{distribution sharpening}, whereby the policy distribution becomes excessively peaked and rare-yet-correct trajectories are progressively suppressed, thus diminishing sample diversity. To this end, we promote flat-reward optima in RLVR to smooth the policy distribution. We theoretically link flat reward RLVR to reduced distribution sharpening, and empirically show improved sample diversity while maintaining single-attempt accuracy. Our method, called \texttt{Flat}-RLVR, searches for flat reward regions via sharpness-aware minimization during LLM post-training, combining this with GRPO variants to achieve better trade-offs between sampling efficiency and reasoning capacity.
| Comments: | 9 pages, 8 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.29281 [cs.LG] |
| (or arXiv:2610.29281v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29281 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yeonbum Kim [view email]
[v1]
Wed, 21 Oct 2026 06:35:19 UTC (644 KB)
来源：arXiv cs.AI · arxiv.org