arXiv cs.AI· Nathaniel Bottman, Alexei Oblomkov·· 1 天前AI 评分31
诚实是一场竞赛：预测可验证奖励 RL 何时教会模型虚张声势
Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff
AI 导读
研究揭示可验证奖励强化学习中的“诚实竞赛”：当检查器接受率与奖励比满足 p̃r/b>1+κ 时，GRPO 单步会让模型更诚实，阈值附近训练结果取决于这场竞赛。在 Datalog 与代码模型实验中，pr/b=1 以上的运行均未崩溃，而 pr/b=3 时两个代码模型把约一半正确程序判为有 bug。深度 3 Datalog 上先宽松训练再削减奖励，三个随机种子下模型至少保持 67% 诚实。
正文 · 原文
Abstract:Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places, with an honest model or one that bluffs. We study rewards that pay $b$ for a verdict that a checker cannot verify ("this program is correct") and $r$ for a certificate that it accepts (an input revealing a bug). To first order, a GRPO step makes the model more honest exactly when $\tilde{p}\,r/b>1+\kappa$. Here $\tilde{p}$ is the rate at which the checker accepts the model's certificates, weighted toward the items on which the model still gives both verdicts, and $\kappa$ is the pull of the items on which the unverifiable verdict is right. Both terms move during training. A checker raises $\tilde{p}$, so near the threshold honesty is a race, which a run just below the threshold can win and a run under a lottery (random pay with the same mean) cannot. The term $\kappa$ starts large and falls as the other items settle, so a run just above the threshold first dips toward bluffing, on code nearly to zero, before climbing back. Paying the honest verdict also pulls the other items toward it, which predicts a ceiling on the payoff ratio, above which the model gives the honest verdict where it is wrong. On propositional Datalog, two code models, and abstention on grade-school math, with $p$ the model's success rate before RL, no run above $pr/b=1$ ends collapsed, and runs under the checker can survive slightly below it. At $pr/b=3$, above the ceiling, both code models call about half of correct programs buggy. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, at a ratio where the same model trained from the start falls to 4-15% and is at most 47% honest after 160 steps. On depth-2 Datalog, when honesty collapses, a linear probe fit before RL still reads the truth.
| Comments: | 34 pages, 10 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.15779 [cs.LG] |
| (or arXiv:2611.15779v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15779 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nathaniel Bottman [view email]
[v1]
Wed, 11 Nov 2026 17:54:41 UTC (1,403 KB)
来源：arXiv cs.AI · arxiv.org