arXiv cs.AI· Sihao Ding, Santosh Vasa, Aditi Ramadwar, Thomas Monninger·· 4 小时前AI 评分40
研究揭示 VLM 忠实性缺口:EDCT 反事实测试与 EDCT-Bench 基准
Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing
AI 导读
研究者提出 Explanation-Driven Counterfactual Testing(EDCT)协议,提取 VLM 解释中引用的视觉概念并施加经核验的最小编辑,检验答案与解释是否仍与编辑后图像一致。
正文 · 原文
Abstract:Vision-Language Models (VLMs) can produce Natural Language Explanations (NLEs) that sound plausible yet remain inconsistent with the visual evidence they cite. We present Explanation-Driven Counterfactual Testing (EDCT), an intervention-based protocol that extracts visual concepts cited in a model's explanation, applies verified minimal edits to them, and tests whether the resulting answer and explanation remain consistent with the edited image. Using this protocol, we create EDCT-Bench, a comprehensive benchmark spanning three complementary domains: knowledge-intensive visual question answering (OK-VQA), safety-critical driving (DriveLM), and 3D spatial reasoning (3DSRBench). Across the evaluated VLMs, EDCT reveals substantial faithfulness gaps, with models frequently producing responses inconsistent with verified visual changes. Finally, our fine-tuning study suggests that EDCT-generated counterfactuals provide high-impact training signals.
| Comments: | Accepted to WACV 2027. 16 pages, including appendix |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.17953 [cs.CV] |
| (or arXiv:2609.17953v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.17953 arXiv-issued DOI via DataCite |
Submission history
From: Santosh Vasa [view email]
[v1]
Wed, 16 Sep 2026 00:25:31 UTC (4,157 KB)
[v2]
Tue, 17 Nov 2026 16:34:12 UTC (4,141 KB)
来源:arXiv cs.AI · arxiv.org