arXiv cs.CL· Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, Cai Yuchen, Yangkun Chen, Weijie Liu, Hao Chen, Yang Wang, Saiyong Yang, Can Yang·· 2 天前AI 评分24
Composition-RL：为大语言模型强化学习组合可验证提示词
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
AI 导读
针对 RLVR 训练中通过率为 1 的简单提示词随训练推进不断增多、压缩有效数据规模的问题，研究者提出离线数据增强方法 Composition-RL，在 RL 之前组合现有提示词，且不修改 RL 优化器。在 4B 至 30B 模型上的实验显示，其推理能力持续优于用原始数据集训练的 RL，课程变体可进一步提升表现，并支持跨域提示词组合。代码、数据与模型均已公开。
正文 · 原文
Authors:Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, Cai Yuchen, Yangkun Chen, Weijie Liu, Hao Chen, Yang Wang, Saiyong Yang, Can Yang
Abstract:Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, an offline data-augmentation approach that composes existing prompts before RL to address the pass-rate-1 bottleneck without modifying the RL optimizer. The composed dataset is generated and verified once from the fixed source pool and then used for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. All our code, data, and models are publicly available to facilitate future research.
| Comments: | NeurIPS 2026 camera-ready version. 33 pages |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2602.12036 [cs.CL] |
| (or arXiv:2602.12036v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2602.12036 arXiv-issued DOI via DataCite |
Submission history
From: Xin Xu [view email]
[v1]
Thu, 12 Feb 2026 15:03:37 UTC (541 KB)
[v2]
Wed, 22 Apr 2026 12:48:11 UTC (624 KB)
[v3]
Thu, 15 Oct 2026 09:59:17 UTC (612 KB)
来源：arXiv cs.CL · arxiv.org