arXiv cs.AI· Chenxing Wei, Jiazhen Kang, Hong Wang, Jianqing Zhang, Hao Jiang, Xiaolong Xu, Ningyuan Sun, Ying He, F. Richard Yu, Yao Shu, Bo Jiang·· 13 小时前AI 评分27
LFPO：面向掩码扩散模型的 token 级策略优化与速度对齐
Efficient Token-Level Policy Optimization for Masked Diffusion Models via Velocity Alignment
AI 导读
研究者提出 Sequence Likelihood-Free Policy Optimization（LFPO），将扩散大语言模型（dLLM）的策略对齐转化为离散速度对齐，绕过序列级似然估计，用对比式 token 级更新直接优化去噪 logits。在推理与代码基准上，LFPO 的准确率匹配或超过基于似然的基线，同时显著加速推理，并可用更少的去噪迭代完成高质量采样。
正文 · 原文
Authors:Chenxing Wei, Jiazhen Kang, Hong Wang, Jianqing Zhang, Hao Jiang, Xiaolong Xu, Ningyuan Sun, Ying He, F. Richard Yu, Yao Shu, Bo Jiang
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) successfully aligns autoregressive models in rigorous domains like mathematical reasoning and code generation. However, applying RLVR to Diffusion Large Language Models (dLLMs) is hindered by intractable sequence-level likelihoods, forcing existing methods to rely on unstable, high-variance approximations. We propose Sequence Likelihood-Free Policy Optimization (LFPO), an efficient framework that circumvents this bottleneck by formulating policy alignment as discrete velocity alignment. Inspired by continuous flow matching, LFPO eschews trajectory-level estimation and directly optimizes denoising logits via contrastive token-level updates, providing a stable, low-variance surrogate. Furthermore, predicting terminal solutions from intermediate states implicitly straightens the generation trajectory, enabling high-quality sampling with significantly fewer denoising iterations. Extensive experiments on reasoning and coding benchmarks demonstrate that LFPO matches or exceeds the accuracy of likelihood-based baselines while substantially accelerating inference.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.01563 [cs.LG] |
| (or arXiv:2603.01563v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.01563 arXiv-issued DOI via DataCite |
Submission history
From: Chenxing Wei [view email]
[v1]
Mon, 2 Mar 2026 07:42:55 UTC (272 KB)
[v2]
Sun, 27 Sep 2026 05:00:38 UTC (300 KB)
[v3]
Tue, 27 Oct 2026 16:51:22 UTC (300 KB)
来源：arXiv cs.AI · arxiv.org