arXiv cs.AI· Ziqi Wen, Parsa Madinei, Miguel P. Eckstein·· 3 天前AI 评分32
研究揭示 VLM 与人类场景描述之间的语义显著性差距
Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency
AI 导读
研究者提出 Counterfactual Semantic Saliency（CSS）框架，通过因果消融物体测量语义偏移，以黑盒方式量化物体重要性。基于 307 个复杂自然场景、1,306 个反事实变体和 16,603 份人类有效回答的对比测试显示，VLM 相比人类更依赖大物体、图像中心物体和高显著性物体，而对场景中人物的依赖更少，其中尺寸偏差是模型与人类语义分歧的主要驱动因素。
正文 · 原文
Abstract:Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Traditional white-box interpretability methods are inapplicable to closed-source architectures and passive metrics fail to isolate causal features. We introduce Counterfactual Semantic Saliency (CSS). This black-box, model-agnostic framework quantifies the importance of objects by measuring the semantic shift induced by their causal ablation from a scene. To evaluate AI-human semantic alignment, we tested prominent VLMs against a human psychophysics baseline comprising 16,603 valid responses across 307 complex natural scenes and 1,306 high-fidelity counterfactual variants. Our analysis reveals a pervasive scene comprehension gap: models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias), and high-saliency objects. In contrast, models rely less on people in the scenes than our human participants do to describe the images. A model's size bias is a primary driver explaining variations in model-human semantic divergence. Code and data are available at this https URL .
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.13047 [cs.CV] |
| (or arXiv:2605.13047v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.13047 arXiv-issued DOI via DataCite |
Submission history
From: Ziqi Wen [view email]
[v1]
Wed, 13 May 2026 06:11:47 UTC (5,479 KB)
[v2]
Sat, 10 Oct 2026 19:45:17 UTC (6,089 KB)
来源：arXiv cs.AI · arxiv.org