arXiv cs.AI· Hongeun Kim, Jeongsol Kim, Jong Chul Ye·· 12 小时前AI 评分31
LeNO：为少步生成模型学习噪声优化实现奖励对齐
Learning Noise Optimization for Reward Alignment in Few-Step Generative Models
AI 导读
LeNO 将奖励引导的初始噪声优化摊销为学习式残差传输，预训练生成器保持冻结，用 Local GRPO 和标量奖励学习以锚点为条件的噪声修正分布，推理时无需迭代搜索。针对高斯初始噪声与保范投影，作者推导出修正噪声分布与原始先验之间 KL 散度的可处理上界，作为正则化目标。在少步文本生图与蛋白质结构生成任务上，LeNO 仅需单次噪声修正即可提升目标奖励，同时基本保持生成质量与多样性。
正文 · 原文
Abstract:Reward alignment for diffusion- and flow-based generative models is often formulated over stochastic trajectories, making it less suitable for few-step deterministic generators. Initial noise optimization provides a trajectory-free alternative, but existing gradient-free methods require costly per-sample search at inference. We introduce LeNO (Learning Noise Optimization), which amortizes reward-guided initial noise optimization into a learned residual transport while keeping the pretrained generator frozen. LeNO learns an anchor-conditioned distribution over noise corrections using Local GRPO and scalar rewards, eliminating iterative test-time optimization. For Gaussian initial noise with norm-preserving projection, we derive a tractable upper bound on the KL divergence between the corrected noise distribution and the original prior, providing a principled regularization objective. We evaluate LeNO on few-step text-to-image and protein structure generation, including expensive black-box designability rewards. Across both domains, LeNO improves target rewards with a single noise correction at inference while largely preserving generation quality and diversity.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.06255 [cs.LG] |
| (or arXiv:2611.06255v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06255 arXiv-issued DOI via DataCite |
Submission history
From: Jeongsol Kim [view email]
[v1]
Wed, 4 Nov 2026 10:06:32 UTC (7,079 KB)
来源：arXiv cs.AI · arxiv.org