arXiv cs.CL· Haruto Yoshida, Keito Kudo, Yoichi Aoki, Ryota Tanaka, Itsumi Saito, Keisuke Sakaguchi, Kentaro Inui·· 11 小时前AI 评分31
信号存在但预测失败：探究大型视觉语言模型中的图表表征
Signals Are Present, Predictions Fail: Probing Diagram Representations in Large Vision-Language Models
AI 导读
基于有向图合成数据集的线性探测实验显示，大型视觉语言模型（LVLM）的隐藏状态中，节点与边的信息可被线性解码，包括模型在 VQA 中难以处理的边方向和节点连通性。这表明模型内部信号与输出之间存在差距。对隐藏状态的因果干预实验进一步表明，部分经探测识别的信号对模型最终预测有因果贡献，且该结论在流程图数据上同样成立。
正文 · 原文
Abstract:Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle to understand relationships between elements, particularly those represented by directed edges between nodes. To investigate the causes of this limitation, we probe the internal representations of LVLMs using a synthetic diagram dataset based on directed graphs. Our linear probing experiments reveal that signals about nodes and edges are linearly decodable from the hidden states of LVLMs, even for aspects that LVLMs struggle with in VQA, such as edge direction and connectivity between nodes. These results reveal a gap between internal signals and model outputs. We further perform causal interventions on the model's hidden states, showing that, for some aspects, the signals identified through probing causally contribute to the model's final predictions. These findings also hold on flowcharts, not only on synthetic diagrams.
| Comments: | Accepted to EMNLP 2026 Main; 38 pages, 58 figures, 18 tables |
| Subjects: | Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2603.02865 [cs.CL] |
| (or arXiv:2603.02865v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.02865 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (2026), pp. 61160-61197 |
| Related DOI: | https://doi.org/10.18653/2026.emnlp-main.2689
DOI(s) linking to related resources |
Submission history
From: Haruto Yoshida [view email]
[v1]
Tue, 3 Mar 2026 11:17:31 UTC (5,600 KB)
[v2]
Fri, 6 Nov 2026 12:53:33 UTC (9,640 KB)
来源：arXiv cs.CL · arxiv.org