Gibbs-Gap Policy Optimization：面向可验证奖励离线强化学习的 GGPO 方法

View PDF HTML (experimental)

Abstract:Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy. Correcting this mismatch with learner-to-behavior importance weights can introduce high variance for long responses. We introduce Gibbs-Gap Policy Optimization (GGPO), which minimizes a log-mean-exp contrast of KL-adjusted rewards among responses to the same prompt, without learner-to-behavior importance weights. We derive GGPO from KL-regularized policy optimization: on policy, its population loss is exactly the KL-regularized optimality gap, and under suitable coverage the learner's single-response expected reward is at least the behavior policy's best-of-$K$ reward, minus terms for the raw finite-group loss, regularization, and finite training data. We characterize reference refresh through exact Gibbs-target composition and a finite-data final-policy reward bound under local reference control and tokenwise behavior mismatch. This refresh is empirically important: in an ablation, fixed-reference GGPO scores $32.96$ on the five-benchmark problem-weighted average, versus $46.58$ with the default refresh schedule. Experiments on mathematical reasoning further show that GGPO outperforms GRPO and AGRO under training-inference mismatch and on fixed offline data, while maintaining its performance as controlled policy staleness grows.

Comments:	51 pages. Code: this https URL
Subjects:	Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as:	arXiv:2610.20814 [cs.LG]
	(or arXiv:2610.20814v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.20814 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xuanchang Zhang [view email]
[v1] Wed, 14 Oct 2026 18:59:29 UTC (432 KB)