arXiv cs.AI· Abhijeet Sinha, Sundari Elango, Dianbo Liu·· 12 小时前AI 评分38
期望回报导致强化学习结果级模式坍缩，逆概率缩放可修复
Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling
AI 导读
研究发现结果级模式坍缩是期望回报目标本身的结构性后果：理想学习动态下，任意两个结果的 log 概率比随奖励差线性演化，导致指数级分化，与探索策略、熵正则和优化算法无关。作者提出逆概率缩放，去除期望中的概率乘子，可证明得到与奖励成比例的终端分布。将其落地为 GRPO 的即插即用改动 IPS-GRPO，无需辅助模型或架构改动，在推理与分子生成任务上持续减少模式坍缩且性能持平或更优。
正文 · 原文
Abstract:Many reinforcement learning (RL) problems admit multiple terminal solutions of comparable quality, where the goal is not to identify a single optimum but to represent a diverse set of high-quality outcomes. Nevertheless, policies trained by standard expected return maximization routinely collapse onto a small subset of outcomes, a phenomenon commonly attributed to insufficient exploration or weak regularization. We show that this explanation is incomplete: outcome level mode collapse is a structural consequence of the expected-return objective itself. Under idealized learning dynamics, the log-probability ratio between any two outcomes evolves linearly in their reward difference, implying exponential ratio divergence and inevitable collapse independent of the exploration strategy, entropy regularization, or optimization algorithm. We identify the source of this pathology as the probability multiplier inside the expectation and propose a minimal correction: inverse probability scaling, which removes outcome-frequency amplification from the learning signal, fundamentally changes the learning dynamics, and provably yields reward-proportional terminal distributions, preventing collapse in multimodal settings. We instantiate this principle in Group Relative Policy Optimization (GRPO) as a drop-in modification, IPS-GRPO, requiring no auxiliary models or architectural changes. Across different reasoning and molecular generation tasks, IPS-GRPO consistently reduces outcome-level mode collapse while matching or exceeding baseline performance, suggesting that correcting the objective rather than adding exploration heuristics is key to reliable multimodal policy optimization.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.21669 [cs.LG] |
| (or arXiv:2601.21669v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.21669 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), Proceedings of Machine Learning Research, vol. 306, pp. 113750-113765, 2026 |
Submission history
From: Abhijeet Sinha [view email]
[v1]
Thu, 29 Jan 2026 13:03:33 UTC (3,186 KB)
[v2]
Tue, 20 Oct 2026 03:12:50 UTC (292 KB)
来源：arXiv cs.AI · arxiv.org