arXiv cs.AI· Junyi Wu, Weijian Luo, Haoyang Zheng, Ruizhe Zhang, Guang Lin·· 10 小时前AI 评分38
DIDR：基于扩散奖励的单步生成器强化学习对齐框架
Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL
AI 导读
研究者提出 Diff-Instruct with Diffused Reward（DIDR），一个从积分 KL 最小化推导的无数据轨迹级对齐框架，将 RLHF 最优的奖励倾斜干净图像分布沿扩散轨迹传播到所有噪声层级。DIDR 在单步 SDXL 基线上实现帕累托占优，迁移到 6B DiT 骨干（Z-Image）后仅需单步生成即超越其 50 步教师模型的偏好对齐表现。
正文 · 原文
Abstract:Recent advances in one-step text-to-image generation have enabled real-time synthesis with remarkable efficiency and quality. Previous reinforcement learning methods for one-step generators combine image-space reward optimization with diffusion noisy-space distribution matching. This paradigm brings challenges due to a mismatch between terminal reward optimization and the underlying generative dynamics. As a result, optimization tends to exploit stochastic degrees of freedom, often improving reward at the expense of image fidelity. To address this issue, we propose Diff-Instruct with Diffused Reward (DIDR), a data-free trajectory-level alignment framework derived from Integral KL minimization. DIDR propagates the RLHF-optimal reward-tilted clean-image distribution across all noise levels along the diffusion trajectory. We show that this objective admits the same minimizer as clean-image RLHF, while naturally inducing the Diffused Reward Score (DRS), which acts as a reward-driven correction to the reference score function. To make this practical, we further introduce the Diffused Reward Proxy (DRP), an efficient estimator of DRS based on differentiable short-step denoising. Extensive experiments demonstrate that DIDR consistently Pareto-dominates existing one-step SDXL baselines. Moreover, when transferred to a 6B DiT backbone (Z-Image), DIDR surpasses its 50-step teacher in preference alignment while requiring only a single generation step. Code: this https URL
| Comments: | 33 pages. Updated manuscript and author information |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.24001 [cs.CV] |
| (or arXiv:2605.24001v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.24001 arXiv-issued DOI via DataCite |
Submission history
From: Junyi Wu [view email]
[v1]
Mon, 18 May 2026 21:26:33 UTC (22,870 KB)
[v2]
Tue, 26 May 2026 04:35:31 UTC (22,871 KB)
[v3]
Tue, 10 Nov 2026 18:16:41 UTC (22,870 KB)
来源：arXiv cs.AI · arxiv.org