RLVR 采样器不匹配下的熵漂移符号定律

View PDF HTML (experimental)

Abstract:Entropy in reinforcement learning with verifiable rewards (RLVR) moves up or down depending on choices that should be neutral, such as whether over-length responses are masked or counted as failures. We trace this to sampler mismatch, where rollouts are drawn with top-k/top-p truncation or at a temperature the learner does not apply, and prove that a constant advantage shift then changes the entropy of a softmax policy, at first order and in expectation, with a sign fixed by the sampler: positive shifts sharpen the policy under any sampler whose likelihood ratio to the policy increases with token probability, including top-k, top-p, min-p, and temperatures below one, and flatten it under flatter samplers. In group-based RLVR with token-level losses, the baseline-controlled part of this drift is set by the token-weighted net advantage, which masking and failing over-length responses can push to opposite signs; the remainder, a covariance between advantages and the sampler's sharpness gap, can feed back over long horizons. On a 2B model, the entropy slope is linear in shifts of the net advantage under truncated sampling and indistinguishable from flat under exact sampling, and token-level importance ratios leave 98% of the drift. In 100-step runs, a net-zero baseline only delays an entropy explosion that exact sampling does not show, a fixed positive shift delays it further, and a baseline weighted by the sharpness gap keeps entropy stable without changing the learner. On a 9B reasoning model over ten steps, the uncorrected rule loses held-out accuracy, whereas zeroing the net advantage or renormalizing the learner gains 8.2 and 8.8 points over it, with no held-out effect under exact sampling.

Submission history

From: Zhiyuan Wang [view email]
[v1] Sun, 25 Oct 2026 04:11:38 UTC (159 KB)