arXiv cs.AI· Qiuyu Kong, Zanxi Ruan, Marco Cristani, Yiming Wang·· 12 小时前AI 评分22
RoSA：缓解显著性坍缩的鲁棒显著性感知长文本图文对齐框架
Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment
AI 导读
研究者提出 RoSA 显著性感知微调框架，通过将图文对分解为显著视图与上下文视图联合学习，缓解模型过度依赖视觉显著文本、忽视非显著上下文的“显著性坍缩”问题。该框架用目标级视觉显著性估计结合 LLM 文本分解实现模态无关的分解策略，并引入可插拔辅助目标显式对齐两种视图。RoSA 显著提升长文本检索表现，在多项基准上对显著性坍缩展现强鲁棒性，代码与 checkpoint 已公开。
正文 · 原文
Abstract:Long-text image-text alignment is essential for fine-grained scene understanding, yet existing methods mainly focus on extending context length or improving local alignment. We identify an overlooked failure mode, saliency collapse, where models over-rely on text that features visual prominence while underutilizing non-salient context. Consequently, retrieval performance degrades sharply when salient content is reduced, despite that the non-salient context remains informative. To address this, we propose RoSA, a saliency-aware finetuning framework that decomposes image-text pairs into complementary salient and contextual views, enabling joint learning of prominent and contextual semantics. We propose a modality-independent decomposition strategy to combine object-level visual saliency estimation with LLM-based text decomposition to bridge vision and language. We employ auxiliary objectives to explicitly align these views, ensuring robust representation with both salient and contextual features. RoSA greatly improves long-text retrieval and demonstrates strong robustness to saliency collapse across benchmarks. Moreover, our auxiliary objectives are plug-and-play, bringing consistent gains to existing tuning methods. Code and checkpoints are available at: this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.31005 [cs.AI] |
| (or arXiv:2610.31005v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31005 arXiv-issued DOI via DataCite |
Submission history
From: Qiuyu Kong [view email]
[v1]
Thu, 22 Oct 2026 09:44:57 UTC (32,220 KB)
[v2]
Fri, 23 Oct 2026 09:09:44 UTC (32,220 KB)
来源：arXiv cs.AI · arxiv.org