arXiv cs.AI· Jinmei Liu, Haoru Li, Zhenhong Sun, Chaofeng Chen, Yatao Bian, Bo Wang, Daoyi Dong, Chunlin Chen, Zhi Wang·· 13 小时前AI 评分31
DRIFT：缓解强化微调中多样性坍缩，提升图像生成多样性
Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation
AI 导读
针对强化微调中策略坍缩为 Dirac delta 分布的多样性坍缩问题，研究者提出 DRIFT 框架，从采样、提示词和优化三方面激励输出多样性。实验显示 DRIFT 在任务对齐与生成多样性上呈明显帕累托优势：对齐相同时多样性提升 3.88%~81.42%，多样性相同时对齐提升 14.38%~56.45%。代码已开源。
正文 · 原文
Abstract:Reinforcement learning (RL) has emerged as a paradigm for fine-tuning large-scale generative models, such as diffusion and flow models, to align with complex human preferences and user-specified tasks. A fundamental limitation remains \textit{the curse of diversity collapse}, where the objective formulation and optimization landscape inherently collapse the policy to a Dirac delta distribution. To address this challenge, we propose \textbf{DRIFT} (\textbf{D}ive\textbf{R}sity-\textbf{I}ncentivized Reinforcement \textbf{F}ine-\textbf{T}uning for Versatile Image Generation), an innovative framework that systematically incentivizes output diversity throughout the on-policy fine-tuning process, reconciling strong task alignment with high generation diversity to enhance versatility essential for applications that demand diverse candidate generations. We approach the problem across three representative perspectives: i) \textbf{sampling} a reward-concentrated subset that filters out reward outliers to prevent premature collapse; ii) \textbf{prompting} with stochastic variations to expand the conditioning space, and iii) \textbf{optimization} of the intra-group diversity with a potential-based reward shaping mechanism. Experimental results show that DRIFT exhibits clear Pareto dominance in task alignment and generation diversity, achieving 3.88\%$\sim$81.42\% higher diversity at matched alignment and 14.38\%$\sim$56.45\% higher alignment at matched diversity. Our code is available at \textcolor{magenta}{\href{this https URL}{this https URL}}.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.12401 [cs.LG] |
| (or arXiv:2601.12401v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.12401 arXiv-issued DOI via DataCite |
Submission history
From: Jinmei Liu [view email]
[v1]
Sun, 18 Jan 2026 13:25:43 UTC (18,311 KB)
[v2]
Wed, 5 Aug 2026 11:59:05 UTC (26,051 KB)
[v3]
Tue, 27 Oct 2026 13:20:56 UTC (28,703 KB)
来源：arXiv cs.AI · arxiv.org