arXiv cs.CL· Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, Linfeng Song, Dian Yu, Xiangliang Zhang, Haitao Mi, Dong Yu·· 14 小时前AI 评分38
EVOL-RL：无标签语言模型自进化中的选择与多样性平衡
Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
AI 导读
针对无标签自改进中"选择诱导的多样性崩溃"问题，研究者提出 EVOL-RL，用多数投票提供稳定选择信号、嵌入新颖度细化信用分配，并以熵正则和非对称裁剪维持多样化采样。在 AIME24 上无标签训练 Qwen3-4B-Base 后，held-out AIME25 的 pass@1 从 TTRL 的 4.6% 提升至 17.1%，pass@16 从 18.5% 提升至 42.0%。代码已开源。
正文 · 原文
Abstract:Large language models (LLMs) can improve through reinforcement learning with verifiable rewards (RLVR), but many settings lack reference answers or external verifiers. Label-free self-improvement instead constructs training signals from the model's own outputs. Because the same policy generates both the candidate responses and the signals derived from them, each update also changes the candidates available for future training, creating a self-reinforcing feedback loop. We identify a failure mode of this loop, which we call selection-induced diversity collapse: repeated self-selection can progressively narrow the candidate pool and make alternative correct solutions harder to sample. To address this problem, we propose EVOL-RL, which balances selection and variation. Majority voting provides a stable selection signal, while embedding-based novelty refines credit among responses with the same selection status; entropy regularization and asymmetric clipping further support this process by maintaining diverse rollouts and preserving rare, high-advantage updates. Across multiple training distributions, model scales, backbones, and matched label-free baselines, EVOL-RL consistently improves pass@n while generally maintaining or improving pass@1. For example, when trained without labels on AIME24, Qwen3-4B-Base improves over TTRL on held-out AIME25 from 4.6% to 17.1% in pass@1 and from 18.5% to 42.0% in pass@16. Training analyses further show that when the current majority is wrong, EVOL-RL better preserves alternative correct solutions. The code is available at: this https URL.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2509.15194 [cs.LG] |
| (or arXiv:2509.15194v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.15194 arXiv-issued DOI via DataCite |
Submission history
From: Yujun Zhou [view email]
[v1]
Thu, 18 Sep 2025 17:50:04 UTC (274 KB)
[v2]
Wed, 1 Oct 2025 05:29:42 UTC (454 KB)
[v3]
Wed, 18 Feb 2026 00:51:43 UTC (488 KB)
[v4]
Sat, 24 Oct 2026 05:17:29 UTC (544 KB)
来源：arXiv cs.CL · arxiv.org