arXiv cs.CL· Khang Nhat Hoang Vo, Belal Shoer, Abhirama Subramanyam Penamakuri·· 14 小时前AI 评分24
邻居的低语：重新审视标题共识作为视觉-语言检索的推理时证据
Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval
AI 导读
冻结编码器在推理时融合图像相似度与图像自身标题的相似度，在九个冻结编码器上使文本到图像 R@1 提升 3.4 至 16.8 点，前提是标题与查询来自同一标注者。
正文 · 原文
Abstract:An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still judge every image-caption pair in isolation. We ask what these neighboring captions contribute when a frozen encoder consults them at inference time. Fusing image similarity with similarity to an image's own captions improves text-to-image R@1 by 3.4 to 16.8 points across nine frozen encoders when the captions come from the same annotators as the queries. The gain falls to 1.9 to 7.2 points when an off-the-shelf captioner writes them, and to at most one point when they come from a different annotation source; a paraphrase-trained text encoder strengthens the text side further. A natural refinement, scoring the part of the image that the captions leave unexplained, appears to add much more under a leave-one-out evaluation. We show that this advantage arises because the correct candidate's supports are built differently from those of the negative candidates, with set size and query removal each contributing about half, and that under a fixed support index the residual reduces to a reweighting of what the captions already provide. On compositional benchmarks, the captions help by matching candidates to human descriptions of the image, without improving image-text grounding. The code is available at: this https URL
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.33798 [cs.CV] |
| (or arXiv:2610.33798v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33798 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Khang Vo Hoang Nhat [view email]
[v1]
Sat, 24 Oct 2026 16:59:25 UTC (7,497 KB)
来源：arXiv cs.CL · arxiv.org