arXiv cs.AI· Michael Li, Rohan Pandey, Ray Chen·· 10 小时前AI 评分31
研究揭示 RLVR 中跨提示共享如何同时抑制与放大奖励黑客
Wrong the Same Way: Cross-Prompt Sharing Both Suppresses and Amplifies Reward Hacking in RLVR
AI 导读
一项受控研究发现，RLVR 在 KL 正则化最优处只对接受的回答集合重新加权而不改变其构成，因此奖励黑客率被基座模型精确继承。在参数共享下，被多数提示接受的错误答案会被放大、仅少数提示接受的则被压到基座率以下，Adam、SGD 和采样式 GRPO 在各收敛学习率下均呈现这一方向。
正文 · 原文
Abstract:Two accounts of how reinforcement learning with verifiable rewards (RLVR) interacts with an imperfect verifier coexist: in one, reward hacking is inherited from the base model; in the other, RL amplifies it. We study both in a controlled environment small enough to compute the entire response distribution exactly. We vary the universality of a leak, or the fraction of prompts that accept the same wrong response, and the verifier's static false-positive rate. At its KL-regularized optimum, RLVR acts as a filter that reweights the accepted set without altering its composition, so the hack rate is inherited exactly. Taking that as a null model, a known decomposition writes the drift from it as a sum over prompts of covariances between reward and the empirical tangent kernel. The terms from other prompts vanish without parameter sharing. With sharing, every prompt that accepts a wrong answer pushes it up and every prompt that rejects it pushes it down through the baseline, in proportion to the model's probability of producing it there. A shared-parameter policy thus drives a leak that few prompts accept below the base rate and amplifies one that most prompts accept. At every convergent learning rate under Adam, SGD, and sampled GRPO, hacking ends higher at high universality than at low, but whether either end crosses the base rate depends on the step size. Independent per-prompt copies remove the dependence. The direction of this push depends on neither the verifier's error rate nor universality alone, but on how much of the model's kernel-weighted probability of a wrong answer falls on prompts that accept it, which frozen base model controls support. Verifier fixes should target the mistakes the model repeats across prompts: two patches that remove equally many errors differ by up to 38 points of true accuracy. We propose this alignment as a hypothesis for why earlier studies disagree.
| Comments: | Accepted at the Transitioning from Pre-Training to Post-Training Workshop at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13218 [cs.LG] |
| (or arXiv:2611.13218v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13218 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Michael Li [view email]
[v1]
Mon, 9 Nov 2026 20:39:36 UTC (2,200 KB)
来源：arXiv cs.AI · arxiv.org