arXiv cs.CL· Sugam Karki, Ra\'ul V\'azquez, Yova Kementchedjhieva·· 12 小时前AI 评分29
视觉 Transformer 的跨模态对齐与表征层级：分层解析文本理解能力
Reading Between the Layers: Cross-Modal Alignment and Representational Hierarchy in Vision Transformers
AI 导读
研究系统分析了基于 Transformer 的视觉编码器如何处理文本及其含义，逐层考察正字法、语法和语义组织。结果显示，强跨模态映射只在最深层出现，且晚于单模态物体与文本识别能力的发展，对应从表层到语义表征的递进。该层级在对比训练模型中最为明显，而更大的自监督模型和文本丰富的训练数据可在无显式语言监督下部分支持类似行为。
正文 · 原文
Abstract:While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment between rendered text and visual concepts together with probing and retrieval-based analyses as a controlled window into these broader capabilities. Our layer-wise analyses span orthographic, grammatical, and semantic organization, and reveal that strong cross-modal mapping emerges only in the deepest layers, after unimodal object and text recognition have already developed, coinciding with a progression from surface-level to semantic representations. Although this hierarchy is strongest in contrastively trained models, larger self-supervised models and text-rich training data partially support similar behaviors without explicit language supervision. Together, our results suggest that increasing model and training-data scale, as well as text-rich training data, can partially compensate for the absence of language supervision.
| Comments: | Accepted to AACL 2026 Findings |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.32463 [cs.CL] |
| (or arXiv:2610.32463v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32463 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sugam Karki [view email]
[v1]
Fri, 23 Oct 2026 10:30:51 UTC (1,112 KB)
来源：arXiv cs.CL · arxiv.org