arXiv cs.AI· Hanpeng Liu, Yaqian Li, Zidan Wang, Shuoxi Zhang, Zonglin Zhao, Zihao Bo, Rinyoichi Takezoe, Kaiwen Long, Kun He·· 6 天前AI 评分22
ITO：面向图像-文本预训练的多视图对齐与训练时融合框架
ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining
AI 导读
ITO 通过多视图图像增强构建多样化跨模态对应关系，并引入轻量训练时多模态融合模块作为正则化器，推理时丢弃该模块以保持双编码器效率。在 100M–1B 图像-文本对规模下，ITO 在相同数据、骨干和算力条件下于分类、检索及多模态基准上持续优于 CLIP，且融合模块的贡献随数据规模增长，能缓解对比学习后期的过拟合。代码已开源。
正文 · 原文
Abstract:Image--text contrastive pretraining has become a dominant paradigm for visual representation learning, yet existing methods often yield representations that remain partially organized by modality rather than by semantics. We propose ITO, a framework addressing this limitation through two complementary mechanisms with distinct roles. Multimodal multiple alignment enriches supervision by constructing diverse cross-modal correspondences from multi-view image augmentations, providing the primary source of discriminative gain. A lightweight training-time multimodal fusion module then acts as a training-time regularizer, encouraging the encoders to produce features that are compatible under fusion. Crucially, the fusion module is discarded at inference, preserving the efficiency of standard dual-encoder architectures while incurring training-time overhead only. Extensive experiments across pretraining scales from millions to billions of image--text pairs show that ITO outperforms strong contrastive baselines at moderate scale and yields consistent gains over CLIP under identical data, backbone, and compute at the 100M--1B scale, on classification, retrieval, and multimodal benchmarks. Controlled comparisons show that these gains are not explained by additional training compute or image augmentation alone. Our analysis further reveals that the contribution of fusion grows with data scale and that it stabilizes optimization, mitigating the late-stage overfitting commonly observed in aggressive contrastive learning. Code is available at this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.02767 [cs.CV] |
| (or arXiv:2603.02767v5 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2603.02767 arXiv-issued DOI via DataCite |
Submission history
From: Hanpeng Liu [view email]
[v1]
Tue, 3 Mar 2026 09:08:53 UTC (6,521 KB)
[v2]
Wed, 4 Mar 2026 01:56:54 UTC (6,521 KB)
[v3]
Mon, 9 Mar 2026 09:27:09 UTC (6,521 KB)
[v4]
Wed, 30 Sep 2026 08:27:16 UTC (6,522 KB)
[v5]
Fri, 9 Oct 2026 01:14:50 UTC (6,523 KB)
来源：arXiv cs.AI · arxiv.org