arXiv cs.CL· Pengfeng Song, Shaozhong Zhang, Pengfei Shao·· 13 小时前AI 评分20
HFCMA：通过动态粒度发现与最优传输实现层次化细粒度跨模态对齐
Hierarchical Fine-Grained Cross-Modal Alignment via Dynamic Granularity Discovery and Optimal Transport
AI 导读
论文提出层次化细粒度跨模态对齐框架 HFCMA，先用动态粒度发现（DGD）将原始图像 patch 和文本 token 聚合为自适应视觉区域与文本短语，再把区域-短语匹配建模为熵正则化最优传输问题，在全局边际约束下估计结构化对齐矩阵。
正文 · 原文
Abstract:Fine-grained image-text retrieval requires a model not only to assess global semantic consistency between an image and a text, but also to model visual evidence for local textual semantics, including objects, attributes, quantities, actions, and relations, so as to distinguish semantically similar candidates. Existing methods commonly use fixed image patches, detected regions, or text tokens as the basic matching units and then aggregate local interactions through max pooling, cross attention, or similarity accumulation. Such strategies remain limited in semantic unit construction and global matching consistency. This paper proposes a hierarchical fine-grained cross-modal alignment framework, termed HFCMA. The framework progressively transforms low-level patch and token representations into adaptive visual regions and textual phrases, and then performs structured region-phrase matching within a global retrieval scoring process. Specifically, HFCMA first adopts dynamic granularity discovery (DGD) to aggregate raw patches and tokens into semantically compact visual regions and textual phrases. It then formulates region-phrase matching as an entropy-regularized optimal transport problem, thereby estimating a structured alignment matrix under global marginal constraints. To reduce background noise and local semantic drift, HFCMA further introduces background debiasing, textual importance weighting, and prototype consistency regularization. Experiments on Flickr30K and MS-COCO show that HFCMA effectively combines dynamic semantic unit construction with OT-based structured matching under a dual-encoder retrieval framework. The resulting fine-grained score provides structured local alignment information for candidate ranking and improves the discrimination of subtle semantic differences.
| Comments: | 8 figures, 6 tables. Code available at this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.30912 [cs.CV] |
| (or arXiv:2610.30912v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30912 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pengfeng Song [view email]
[v1]
Thu, 22 Oct 2026 08:33:46 UTC (3,020 KB)
来源：arXiv cs.CL · arxiv.org