arXiv cs.CL· Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi, Linrong Cai, Xingyu Fu, Zhuang Liu·· 6 天前AI 评分31
OccludeBench：VLM 能否找出图中缺失的物体？
OccludeBench: Can VLMs Find What's Missing?
AI 导读
研究团队提出 OccludeBench 基准，用于评测视觉语言模型（VLM）从残留视觉线索中还原被完全遮挡物体的能力，并区分需依赖阴影、倒影等物理线索的"直接"案例与依赖场景上下文的"推断"案例。
正文 · 原文
Abstract:How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13790 [cs.CV] |
| (or arXiv:2610.13790v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13790 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Addison J. Wu [view email]
[v1]
Fri, 9 Oct 2026 16:04:54 UTC (7,190 KB)
来源：arXiv cs.CL · arxiv.org