跳到正文
arXiv cs.AI· Sihao Ding, Santosh Vasa, Aditi Ramadwar, Thomas Monninger·· 4 小时前AI 评分40

研究揭示 VLM 忠实性缺口:EDCT 反事实测试与 EDCT-Bench 基准

Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing

AI 导读

研究者提出 Explanation-Driven Counterfactual Testing(EDCT)协议,提取 VLM 解释中引用的视觉概念并施加经核验的最小编辑,检验答案与解释是否仍与编辑后图像一致。

正文 · 原文

View PDF HTML (experimental)

Abstract:Vision-Language Models (VLMs) can produce Natural Language Explanations (NLEs) that sound plausible yet remain inconsistent with the visual evidence they cite. We present Explanation-Driven Counterfactual Testing (EDCT), an intervention-based protocol that extracts visual concepts cited in a model's explanation, applies verified minimal edits to them, and tests whether the resulting answer and explanation remain consistent with the edited image. Using this protocol, we create EDCT-Bench, a comprehensive benchmark spanning three complementary domains: knowledge-intensive visual question answering (OK-VQA), safety-critical driving (DriveLM), and 3D spatial reasoning (3DSRBench). Across the evaluated VLMs, EDCT reveals substantial faithfulness gaps, with models frequently producing responses inconsistent with verified visual changes. Finally, our fine-tuning study suggests that EDCT-generated counterfactuals provide high-impact training signals.
Comments: Accepted to WACV 2027. 16 pages, including appendix
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as: arXiv:2609.17953 [cs.CV]
  (or arXiv:2609.17953v2 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2609.17953

arXiv-issued DOI via DataCite

Submission history

From: Santosh Vasa [view email]
[v1] Wed, 16 Sep 2026 00:25:31 UTC (4,157 KB)
[v2] Tue, 17 Nov 2026 16:34:12 UTC (4,141 KB)

来源:arXiv cs.AI · arxiv.org