arXiv cs.CL· Guangzhi Xiong, Qiao Jin, Sanchit Sinha, Zhiyong Lu, Aidong Zhang·· 10 小时前AI 评分31
重新审视大视觉语言模型胸片推理中的视觉归因
Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
AI 导读
研究提出因果评估框架,仅保留经反事实编辑验证专家标注区域确实对模型预测起因果作用的 CXR-VQA 样本,据此测试 11 种归因方法、6 个开源 LVLM 和两种输出模式,发现现有归因方法常无法识别模型实际使用的视觉证据。为此提出基于概念的归因方法 MedFocus,通过非平衡最优传输定位有临床意义的解剖区域,并用定向干预测量其因果效应,性能大幅优于此前方法。数据与代码已公开。
正文 · 原文
Abstract:Large Vision Language Models (LVLMs) show promise in medical applications, but their inability to faithfully ground responses in visual evidence raises serious concerns about clinical trustworthiness. While visual attribution methods are widely used to explain LVLM predictions, whether these explanations actually reflect the visual evidence underlying the model's decision is largely unverified, since ground-truth annotations for internal model reasoning are typically unavailable. We address this question for chest X-ray (CXR) reasoning by developing a causal evaluation framework that retains only CXR-VQA samples for which the expert-annotated region is verified, via counterfactual editing, to be causally responsible for the model's prediction. Using this framework across 11 attribution methods, six open-source LVLMs, and two output modes (direct answer and step-by-step reasoning), we find that existing attribution methods often fail to identify the evidence used by LVLMs. To address this failure, we propose MedFocus, a concept-based attribution method that localizes clinically meaningful anatomical regions via unbalanced optimal transport and measures their causal effect on model outputs through targeted interventions. MedFocus produces spatial, concept-level, and token-level attributions and substantially outperforms prior methods, taking a step toward more trustworthy attribution for medical LVLMs. Our data and code are available at this https URL.
| Comments: | NeurIPS 2026 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.20158 [cs.CV] |
| (or arXiv:2605.20158v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.20158 arXiv-issued DOI via DataCite |
Submission history
From: Guangzhi Xiong [view email]
[v1]
Tue, 19 May 2026 17:46:40 UTC (3,450 KB)
[v2]
Thu, 19 Nov 2026 18:47:04 UTC (3,455 KB)
来源:arXiv cs.CL · arxiv.org