arXiv cs.CL· Junyi Gao, Yu Shi, Pingzhao Hu, Ewen M Harrison·· 11 小时前AI 评分42
研究揭示 GRPO 在组依赖奖励下变成零和锦标赛，提出 Team-GRPO 修正
When Rewards Depend on the Group, GRPO Plays a Tournament
AI 导读
研究证明 GRPO 的组相对基线在奖励依赖同组其他回答时会改变优化目标，使每个回答被训练成在零和锦标赛中击败组内同伴，K=2 时看不到成对多样性奖励，任意 K 下所见不足一半。
正文 · 原文
Abstract:Rewards for diversity, coverage and distribution matching make the reward of each sampled response depend on the rest of its group, and they are now routinely optimized with GRPO and RLOO. We prove that in this setting the group-relative baseline changes the objective. The expected update equals each response's own reward gradient plus $\alpha$ times its effect on its groupmates' rewards, with $\alpha=-1/(K-1)$ for groups of size $K$, so GRPO trains every response to beat its groupmates in a zero-sum tournament. It cannot see a pairwise diversity bonus when K=2, sees less than half of it at any K, and often does worse than using no baseline at all. Setting $\alpha=+1$ gives Team-GRPO, an unbiased estimator for any group-dependent reward that works with two samples per prompt and requires no extra model calls for pairwise rewards. The theory predicts training quantitatively. It locates the fixed points of a 9B policy to within 0.02 in $\ell_1$ and predicts how concentrated trained policies become with Pearson r=0.995. On GlobalOpinionQA, GRPO collapses onto the majority answer and ends further from human opinion than the untrained model, while Team-GRPO cuts the error from $0.20$ to $0.08$. On NoveltyBench, Team-GRPO avoids the diversity collapse of GRPO and significantly improves set utility.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.02404 [cs.LG] |
| (or arXiv:2611.02404v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02404 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junyi Gao [view email]
[v1]
Mon, 2 Nov 2026 02:26:46 UTC (274 KB)
来源：arXiv cs.CL · arxiv.org