利用 OCR Heads 将图像语义语言化：VLM 视觉语言对齐机制研究

View PDF HTML (experimental)

Abstract:How do VLMs map from pixels to semantics? To understand this general question, we focus on a narrow one: studying how VLMs perform optical character recognition (OCR). Across four models, we identify attention heads causally necessary for OCR, and discover that these are in fact general-purpose heads that output interpretable semantic features across all image tokens. For example, pointing these heads at an image token containing the word "bike" causes Qwen3-VL-8B to output "bike," but pointing them at a bird wing causes the model to output the token "feathers." We collapse these heads' attention weights into a single verbalization lens transformation that reveals interpretable semantic features in hidden states across all layers. When combined with projection to vocabulary space, we can obtain interpretable labels starting from layer 0, showing that image representations are in fact aligned with language in early layers. We find that we can also use the inverse of this transformation to edit non-word concepts, e.g., replacing a tractor with a revolver in a naturalistic image, providing causal evidence that this subspace is useful for more than just OCR. Our results are an example of how the study of specific mechanisms can shed light on broader interpretability problems.

Comments:	21 pages, 22 figures
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
ACM classes:	I.2.7
Cite as:	arXiv:2609.18823 [cs.CV]
	(or arXiv:2609.18823v2 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2609.18823 arXiv-issued DOI via DataCite

Submission history

From: Sheridan Feucht [view email]
[v1] Wed, 16 Sep 2026 15:30:17 UTC (25,383 KB)
[v2] Mon, 2 Nov 2026 16:14:54 UTC (25,382 KB)