arXiv cs.AI· Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti, Zaid Alyafeai, Paul Gavrikov, Bernard Ghanem, Abdelrahman Eldesokey·· 13 小时前AI 评分42
CounterCount：诊断视觉语言模型计数属性时的知识偏差
CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes
AI 导读
研究者推出 CounterCount 诊断数据集，用配对的事实与反事实图像测试视觉语言模型计数属性时的知识偏差。16 个 VLM（14 个开源、2 个闭源）平均准确率从事实图像的 91.6% 降至反事实图像的 49.5%，多数错误给出的是常识性标准数量。分析显示模型在浅层识别主体后即回退到先验知识，其激活中已部分包含正确作答所需的视觉证据却未被充分利用。
正文 · 原文
Abstract:Do Vision-Language Models (VLMs) count what they see, or what they know about the world? We introduce CounterCount, a diagnostic dataset of paired factual and counterfactual (CF) images, in which the CF image contains an atypical count of a subject's attribute (e.g., a rabbit with four ears), together with annotations localizing the counted attributes. Humans answer both types of images nearly perfectly ($98.9\%$ and $97.5\%$), but across 16 VLMs (14 open-weight and 2 closed-source), average accuracy drops from $91.6\%$ on factual to $49.5\%$ on CF images, with most errors stemming from responses that match the canonical count, therefore revealing a bias towards knowledge priors rather than reliance on visual information. Based on these findings, we perform an analysis that reveals two interesting insights behind this gap. First, models exhibit a ``System-1''-like response behavior, falling back on canonical counts after a shallow recognition of the subject. Second, model activations partially contain the visual evidence necessary to answer correctly, yet models underutilize it, attending less to the counted attributes when they provide biased answers. Input manipulations that break visual context, introduce cueing or reasoning, or direct amplification of the model's internal attention to relevant regions can reduce such biases. CounterCount goes beyond showing that VLMs fall back on knowledge priors, revealing that once a familiar context is recognized, models favor prior knowledge over available visual evidence rather than failing to encode that evidence.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.17826 [cs.CV] |
| (or arXiv:2605.17826v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.17826 arXiv-issued DOI via DataCite |
Submission history
From: Reem Alzahrani [view email]
[v1]
Mon, 18 May 2026 04:00:05 UTC (9,311 KB)
[v2]
Tue, 20 Oct 2026 15:34:58 UTC (8,176 KB)
来源：arXiv cs.AI · arxiv.org