arXiv cs.CL· Niclas Lietzow, Danielle Bitterman, Carsten Eickhoff, William Rudman, Michal Golovanevsky·· 12 小时前AI 评分32
视觉优先、先验覆盖：视觉语言模型中感知-知识冲突的因果机制
Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models
AI 导读
研究通过激活修补、组件消融与机制分析发现，视觉语言模型默认以视觉证据为依据，而依赖先验知识的回答由网络中少量因果必要的注意力头（2.5-4.8%，集中于网络后半部分）驱动。
正文 · 原文
Abstract:Vision-language models must reconcile visual evidence with memorized world knowledge when the two conflict. How they resolve this conflict shapes the reliability of multimodal systems, yet prior work characterizes it behaviorally without a component-level causal account. We combine activation patching across three granularities (residual stream, attention heads, and MLP sublayers) with model-component ablation studies and mechanistic analysis. Across three VLM families, we find that visual grounding emerges by default, whereas prior grounding depends on a small set of causally necessary attention heads (2.5-4.8%) concentrated in the second half of the network. These heads enable answers from stored world knowledge (e.g., "red" for a strawberry) despite conflicting visual input. Ablating them flips predictions from knowledge-grounded to visually grounded answers in 68-96% of cases under prior-knowledge prompts, but changes only 0.8-7.5% of visually grounded predictions, establishing an asymmetric causal structure. The identified heads decompose into routing heads, which modulate information flow, and writing heads, which directly project answer tokens into the residual stream. This structure is consistent across model families and scales, revealing a sparse causal circuit underlying perception-knowledge conflict in VLMs.
| Comments: | Findings of EMNLP 2026. 17 pages, 11 figures, 13 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.28273 [cs.CL] |
| (or arXiv:2606.28273v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.28273 arXiv-issued DOI via DataCite |
Submission history
From: Niclas Lietzow [view email]
[v1]
Fri, 26 Jun 2026 17:16:04 UTC (687 KB)
[v2]
Thu, 5 Nov 2026 17:21:24 UTC (712 KB)
来源：arXiv cs.CL · arxiv.org