arXiv cs.CL· Fang Wu, Weihao Xuan, Danny Xing, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi·· 3 天前AI 评分41
RLVR 的隐形缰绳：可验证奖励强化学习能否突破其起源
The Invisible Leash: Why RLVR May or May Not Escape Its Origin
AI 导读
研究提出"经验支持动态"框架，逐题追踪 RLVR 对已验证解法的影响，发现在所评估的模型级基准聚合上，观察到的支持收缩超过扩张——失去已验证成功的问题多于新增的问题，尽管 pass@1 同时上升。部分 RLVR 模型平均 token 级熵升高而最终答案熵降低，token 选择更不确定但答案更集中。这揭示出当前 RLVR 实践存在"隐形缰绳"：让已有成功更可靠，却丢失了其他已验证解法。
正文 · 原文
Abstract:Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, \textit{the shrinkage of observed support outweighs its expansion}: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in \texttt{pass@1}. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2507.14843 [cs.LG] |
| (or arXiv:2507.14843v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2507.14843 arXiv-issued DOI via DataCite |
Submission history
From: Fang Wu [view email]
[v1]
Sun, 20 Jul 2025 07:04:08 UTC (273 KB)
[v2]
Fri, 26 Sep 2025 04:52:44 UTC (415 KB)
[v3]
Wed, 7 Jan 2026 01:59:36 UTC (413 KB)
[v4]
Wed, 4 Feb 2026 00:51:57 UTC (420 KB)
[v5]
Wed, 14 Oct 2026 06:13:12 UTC (485 KB)
来源：arXiv cs.CL · arxiv.org