arXiv cs.AI· Tian Lan, Yiming Zheng, Jianxin Yin·· 14 小时前AI 评分24
Diff-Feat：基于扩散模型的跨模态特征提取用于多标签分类
Diff-Feat: Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification
AI 导读
Diff-Feat 从预训练扩散 Transformer 中提取图像与文本的中间特征并融合，用于多标签分类。图像任务的最优特征位于扩散中间步与 Transformer 中间层，文本任务则位于无噪声步与最深层；该方法在 Visual Genome 500 上达 46.1% mAP，在 Pix2Cap-COCO 上达 80.2% mAP，优于强基线。代码已开源。
正文 · 原文
Abstract:Multi-label classification has broad applications and depends on powerful representations capable of capturing multiple interactions. We introduce Diff-Feat, a simple yet effective framework that extracts intermediate features from pre-trained diffusion-Transformer models for both images and text, and fuses them for downstream tasks. We observe distinct patterns in optimal feature selection: for vision tasks, the most discriminative intermediate feature along the diffusion process occurs at the middle step and is located at the middle block in Transformer. In contrast, for language tasks, the best feature occurs at the noise-free step and is located at the deepest block. Notably, a consistent phenomenon across diverse datasets: a fixed "Middle Layer" yields the best performance on various downstream classification tasks for images (under DiT-XL/2-256$\times$256 and various U-ViT architectures). We further analyze the underlying mechanism and its scaling behavior across model scales. To efficiently select representations, we devise a heuristic local-search algorithm that pinpoints the locally optimal "image-text"$\times$"block-timestep" pair among a few candidates, avoiding an exhaustive grid search. A simple fusion-linear projection followed by addition-of the selected representations achieves 46.1% mAP on Visual Genome 500 (with 500 categories) and 80.2% mAP on Pix2Cap-COCO (with 80 categories), consistently outperforming strong baseline methods. Code is available at this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Applications (stat.AP) |
| Cite as: | arXiv:2509.15553 [cs.CV] |
| (or arXiv:2509.15553v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2509.15553 arXiv-issued DOI via DataCite |
|
| Journal reference: | 2026 International Joint Conference on Neural Networks (IJCNN) |
| Related DOI: | https://doi.org/10.1109/IJCNN69854.2026.11728464
DOI(s) linking to related resources |
Submission history
From: Tian Lan [view email]
[v1]
Fri, 19 Sep 2025 03:13:58 UTC (3,521 KB)
[v2]
Sat, 24 Oct 2026 07:02:30 UTC (6,790 KB)
来源：arXiv cs.AI · arxiv.org