arXiv cs.CL· Shubh Garg, Chen Shani·· 6 天前AI 评分32
研究揭示 VLM 空间关系表征弱于词汇先验：EICS 与 CASP-VL 基准
When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors
AI 导读
研究提出 EICS 指标与基于 Visual Genome 场景图构建的 CASP-VL 基准，测试 VLM 在受控单概念干预下表征是否稳定编码概念。结果显示，多种 VLM 架构对空间关系的表征忠实度显著低于词汇信息，即便在组合类基准上得分远高于随机水平。关系在谓词自身 token 处可识别，但经 EOS token 或均值池化聚合为全局表征后即丢失，形成输出评测无法察觉的系统性失败。
正文 · 原文
Abstract:Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning. We introduce EICS (Evaluation-Invariant Concept Stability), a metric for testing whether representations consistently encode concepts under controlled one-concept interventions, and CASP-VL (Causally controlled Spatial Perturbations for Vision-Language models), a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across multiple VLM architectures, we find that spatial relations are represented substantially less faithfully than lexical information, even when the same models score well above chance on compositional benchmarks. Further analysis shows that the relation is identifiable at the predicate's own tokens but is lost when the sentence is aggregated into a global representation, whether through the (EOS) token or by mean pooling. These findings reveal a systematic failure invisible to output-based evaluation: VLMs do not preserve, in the representation used downstream, relational information that is present in their own input.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13910 [cs.CL] |
| (or arXiv:2610.13910v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13910 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chen Shani [view email]
[v1]
Fri, 9 Oct 2026 17:50:03 UTC (83 KB)
来源：arXiv cs.CL · arxiv.org