arXiv cs.AI· Yue Yu, Bowen Zuo, David Crandall, Yinglun Zhu, Dongruo Zhou·· 6 天前AI 评分22
探索更多、推理更好：面向扩散语言模型的分步风险敏感 GRPO
Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models
AI 导读
研究者提出 StepRS-GRPO，在去噪状态间调整组优势变换的风险系数，同时保留原有训练器。在二值奖励下，该变换等价于对中心化结果优势做提示词与状态相关的重缩放。在多个 dLLM 骨干和数学推理基准上，它相比中心化 GRPO 提升了 pass@1 准确率与 pass@k 覆盖率，并增加了答案多样性。
正文 · 原文
Abstract:Diffusion large language models (dLLMs) generate text by denoising a sequence or successive blocks, allowing several tokens to be revealed in parallel. Reinforcement learning with verifiable rewards (RLVR) reuses terminal feedback across these decisions, even as their conditioning context changes. We propose stepwise risk-sensitive GRPO (StepRS-GRPO), which varies the risk coefficient of the group-advantage transformation across denoising states while retaining the underlying trainer. For binary rewards, we show that this transformation is exactly a prompt- and state-dependent rescaling of centered outcome advantages. A capability-based calibration suggests a coefficient scale, while endpoint and interpolation ablations guide schedule selection. Across multiple dLLM backbones and mathematical reasoning benchmarks, StepRS-GRPO improves both pass@1 accuracy and pass@k coverage over centered GRPO, while increasing answer diversity. In our ablation studies, mass-matched controls support the contributions of state allocation and schedule direction, and the gains persist after matching the root mean square (RMS) of the advantages to that of centered GRPO. Reasoning-trace diagnostics further show that the diversity gains from StepRS-GRPO extend beyond final-answer strings.
| Comments: | 40 pages, 12 figures, 2 tables. The first two authors contributed equally |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.00661 [cs.LG] |
| (or arXiv:2610.00661v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00661 arXiv-issued DOI via DataCite |
Submission history
From: Yue Yu [view email]
[v1]
Wed, 30 Sep 2026 19:59:48 UTC (665 KB)
[v2]
Fri, 9 Oct 2026 07:30:38 UTC (665 KB)
来源：arXiv cs.AI · arxiv.org