arXiv cs.AI· Jiayuan Wang, Tongyao Pang·· 13 小时前AI 评分38
DiffusionRVGE：基于回归值梯度估计的扩散模型强化学习
DiffusionRVGE: Reinforcement Learning via Regression-Based Value Gradient Estimation
AI 导读
DiffusionRVGE 提出一种通过将中心化奖励回归到中心化最终去噪图像来构建更新方向的扩散模型强化学习方法，无需奖励导数，也无需沿采样轨迹做伴随反向传播。
正文 · 原文
Abstract:Reinforcement learning improves diffusion models using rewards on generated images, but repeated denoising and gradient estimation make training expensive. We introduce \method, an efficient method that constructs useful update directions by regressing centered rewards on centered final denoised images. It requires neither reward derivatives nor adjoint backpropagation through the sampling trajectory. A unified first-variation formulation identifies the value-gradient target with or without a trajectory-KL penalty. A population analysis under Gaussian noising explains the regression geometry, while a subspace analysis motivates clean-image coordinates. The implementation adapts rollout bandwidth and query weights to reward continuity: local rollouts suit smooth rewards, whereas broader groups provide informative contrasts for discrete rewards. Trained jointly on five rewards, one SD3.5-Medium model matches or exceeds DiffusionNFT's joint model on seven of eight metrics, reaching GenEval $0.94$, OCR $0.92$, and PickScore $23.82$ at about one fifth of its estimated training compute. In single-reward training, \method reaches matched reward thresholds $2.1$--$6.7\times$ faster than reproduced DiffusionNFT on identical hardware. Ablations support the clean-image estimator, direct direction transfer, and reward-dependent rollout choices. Code is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.22973 [cs.AI] |
| (or arXiv:2610.22973v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22973 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tongyao Pang [view email]
[v1]
Fri, 16 Oct 2026 05:46:50 UTC (6,512 KB)
来源：arXiv cs.AI · arxiv.org