arXiv cs.AI· Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan Manoharan·· 10 小时前AI 评分34
MCPO：按 token 犹豫程度分配奖励的强化学习方法，在数学基准上超越 GRPO
Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards
AI 导读
研究者提出 MCPO（Margin-Credit Policy Optimization），用策略下一 token 分布的 top-2 margin 衡量模型犹豫程度，将 GRPO 平均分配的奖励改为按 token 差异化分配，且无需额外模型评估或批次统计。
正文 · 原文
Abstract:In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin $M=p_1-p_2$ of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95\% CI $[4.1, 9.0]$), and a Qwen3-4B-Base replication improves for all three seeds ($+4.75$). The gain is largest on problems that GRPO solves only some of the time and grows with $k$ in pass@$k$, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14015 [cs.LG] |
| (or arXiv:2611.14015v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14015 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chi Liu [view email]
[v1]
Tue, 10 Nov 2026 12:51:43 UTC (186 KB)
来源：arXiv cs.AI · arxiv.org