arXiv cs.AI· Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, Hui Xiong·· 12 小时前AI 评分32
GVPO：面向大语言模型后训练的分组方差策略优化
GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
AI 导读
研究者提出分组方差策略优化（GVPO），将 KL 约束奖励最大化的解析解直接嵌入梯度权重，使梯度与最优策略对齐。GVPO 保证存在唯一最优解且恰为 KL 约束奖励最大化目标，并支持灵活采样分布，规避 on-policy 与重要性采样的限制。该工作已被 NeurIPS 2025 接收，扩展版 GVPO++ 另覆盖 on-policy 蒸馏。
正文 · 原文
Abstract:Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.
| Comments: | Accepted by NeurIPS 2025. Extended version is available as "GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation" (arxiv.org/abs/2609.21432) |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2504.19599 [cs.AI] |
| (or arXiv:2504.19599v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2504.19599 arXiv-issued DOI via DataCite |
Submission history
From: Kaichen Zhang [view email]
[v1]
Mon, 28 Apr 2025 09:02:24 UTC (23 KB)
[v2]
Mon, 19 May 2025 06:40:29 UTC (873 KB)
[v3]
Mon, 27 Oct 2025 10:30:01 UTC (269 KB)
[v4]
Fri, 16 Oct 2026 08:09:27 UTC (266 KB)
来源：arXiv cs.AI · arxiv.org