arXiv cs.CL· Hongze Chen, Miao Pan, Lizhou Liang, Qinfeng Li, Xinyin Ma, Yongliang Shen, Weishi Wang, Daniel Dahlmeier, Xuhong Zhang, Wenqi Zhang·· 12 小时前AI 评分26
DTCA：面向扩散语言模型强化学习的去噪 Token 信用分配
DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models
AI 导读
针对扩散语言模型迭代去噪过程中信用分配困难的问题，研究者提出 DTCA 框架，通过衡量 token 提交后剩余掩码位置预测置信度的提升来估计 token 贡献，并按词性分组筛选去噪时间步、分配 token 级学习信号。在 LLaDA-8B-Instruct 上，DTCA 在不同生成预算下均提升数学推理表现，MATH500 最高提升 9.0 分，GSM8K 提升 5.3 分。
正文
Abstract:Diffusion language models (DLMs) generate text through iterative denoising, enabling bidirectional context modeling and parallel token refinement. Recent studies have applied reinforcement learning (RL) to DLMs and improved their reasoning performance. However, the iterative denoising process makes credit assignment more challenging, as final outcome rewards provide limited information about how different tokens contribute to the final output. In this paper, we propose Denoising Token Credit Assignment (DTCA), a framework for token-level credit assignment in reinforcement learning for diffusion language models. DTCA estimates token contributions by measuring the improvement in prediction confidence of remaining masked positions after token commitment and groups these signals according to part-of-speech (POS) categories. These contribution weights are used to select informative denoising timesteps and assign token-level learning signals during RL training. DTCA further calibrates these learning signals according to token-level entropy changes induced by policy updates. Experiments on LLaDA-8B-Instruct show that DTCA consistently improves mathematical reasoning performance across different generation budgets, with gains of up to 9.0 points on MATH500 and 5.3 points on GSM8K over the base model.
| Comments: | 20 pages, 10 figures, including appendices |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35201 [cs.CL] |
| (or arXiv:2610.35201v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35201 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongze Chen [view email]
[v1]
Mon, 26 Oct 2026 10:54:11 UTC (1,929 KB)
来源：arXiv cs.CL · arxiv.org