arXiv cs.CL· Junxing Hu, Zicheng Zhang, Zhiqian Zhang, Ai Han, Yifeng Zhang, Pengzhang Liu, Ling Li·· 13 小时前AI 评分31
Info-GRPO：用相关性感知探索解决长推理链训练中的策略崩溃
Correlation-Aware Exploration for Long-Trace Reasoning
AI 导读
针对 GRPO 训练长链推理模型易发生策略崩溃的问题，研究者提出 Info-GRPO，通过潜变量与轨迹间的互信息引导探索，让多样性贯穿整条推理链而非依赖逐 token 随机性。在 AIME24 上，该方法在四个骨干模型上分别将 Avg@8 较 GRPO 提升 3.75%、1.66%、4.16%、4.17%，并以 8 倍更少样本达到相同多数投票准确率。代码与模型将公开。
正文 · 原文
Abstract:Reasoning models trained with Group Relative Policy Optimization (GRPO) now produce chains of thought thousands of tokens long, and recent studies reveal that such training is prone to policy collapse. Entropy regularization is the standard remedy, but it works by pushing probability mass away from the few tokens that carry it and toward the vast low-probability tail of the vocabulary, and it applies this force once for every token of the reasoning trace. This puts the bonus in direct conflict with the entropy decay that convergence requires, and makes its coefficient difficult to set in the long-trace regime. To resolve this tension, we propose Info-GRPO, an information-theoretic framework that cultivates correlation between the policy and a latent prior. Rather than raising entropy everywhere, Info-GRPO builds on the mutual information between a latent variable and the trajectory it induces: augmenting prompts with latent seeds during training lets the model explore diverse policies correlated with the prior, while the seed-conditioned entropy is guided toward convergence. Diversity is therefore carried across whole reasoning traces rather than by per-token randomness, and exploration no longer fights entropy reduction. Experiments show that Info-GRPO outperforms vanilla GRPO and entropy-regularized GRPO across diverse reasoning benchmarks. On AIME24, it improves $Avg@8$ over GRPO by 3.75%, 1.66%, 4.16%, and 4.17% on four backbones, and it reaches the same majority-vote accuracy with $8\times$ fewer samples. Code and models will be publicly released.
| Comments: | 28 pages, 6 figures. Accepted at the Workshop on Long-Context Foundation Models (LCFM), NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34949 [cs.LG] |
| (or arXiv:2610.34949v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34949 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junxing Hu [view email]
[v1]
Mon, 26 Oct 2026 07:19:00 UTC (7,330 KB)
来源：arXiv cs.CL · arxiv.org