arXiv cs.AI· Xiaoyu Wu, Yifei Wang, Weihang Guo, Chen Wei, Lydia E Kavraki, Niloofar Mireshghallah, Steven Wu·· 23 小时前AI 评分38
分层扩散 Transformer(HiT):端到端图像生成新架构
Hierarchical Diffusion Transformer for End-to-End Image Generation
AI 导读
研究者提出分层扩散 Transformer(HiT),在主干网络中逐步将相关 token 分组并用单个 token 表示,从而在生成器内部学习视觉层级。HiT 在 ImageNet 256×256 上达到 1.48 FID,训练 FLOPs 约为 JiT 的 40% 即取得相当生成质量。
正文 · 原文
Abstract:Pixel-space diffusion learns visual representations and image generation jointly in a single end-to-end training stage, without a separately pretrained visual encoder. However, plain pixel-space Transformers still trail leading two-stage latent diffusion models in generation quality and training efficiency. We hypothesize that part of this gap arises from retaining one token per patch throughout the backbone, which does not explicitly encourage a visual hierarchy that summarizes related content at progressively coarser levels. To address this, we introduce Hierarchical Diffusion Transformers (HiT), which progressively group related tokens and represent each group with a single token. The main Transformer bottleneck processes these compact group tokens, and ungrouping restores features to their original image positions. A compression budget regulates the average compression ratio across the batch, allowing group memberships and counts to vary across images and diffusion timesteps during both training and generation. The model exhibits emergent semantic structure, learning to group coherent objects and object parts without segmentation labels. In text-to-image generation, HiT further learns to group caption words with image patches depicting the same concepts, without region annotations. HiT achieves 1.48 FID on ImageNet $256\times256$ and reaches comparable generation quality to JiT using approximately 40\% of its cumulative training FLOPs. For text-to-image generation, HiT exceeds UniLiP's average score across five benchmarks with roughly $6\times$ fewer denoiser parameters and $3\times$ fewer forward FLOPs. These results suggest that learning a visual hierarchy within the generator breaks down the barrier to stronger image generation without external visual pretraining.
| Comments: | Under Review |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.22962 [cs.CV] |
| (or arXiv:2611.22962v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22962 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaoyu Wu [view email]
[v1]
Tue, 17 Nov 2026 15:47:56 UTC (23,409 KB)
来源:arXiv cs.AI · arxiv.org