arXiv cs.AI· Yuyang Dai, Yuxia Wang, Zhuohan Xie·· 12 小时前AI 评分44
视觉语言模型能读对每个事实，却仍给出错误答案：多证据整合缺口研究
All the Facts, Still the Wrong Answer: Integration Gaps in Vision-Language Models
AI 导读
研究提出“多证据整合缺口”概念，考察视觉语言模型能否把多个事实组合成满足全部约束的决策。Gemma-3-27B 能正确恢复所有被评估的孤立条目事实，但在八条目视觉实例上全局有效率仅 19.75%。隐藏状态探针可从全局提示状态解码出解集归属信息，干净状态替换能在依赖模型的深度区间内恢复失败决策，说明恢复事实与表征解相关信息都不足以支撑可靠的联合决策。
正文
Abstract:Multimodal reasoning requires models to combine multiple pieces of evidence into decisions that satisfy all relevant constraints. Evaluations centered on final-answer accuracy leave unclear whether failures arise from reading individual facts or using them together. We study this question through the multi-evidence integration gap, linking local fact recovery to global decision validity on controlled subset-selection and path-finding tasks. Our evaluation combines conditional performance given correct local answers with same-grid local queries, text-oracle inputs, retrieval controls, and constraint ablations. We further examine hidden-state probes and activation replacement across network depth, token positions, and model families. Across the evaluated models, accurate local recovery coexists with substantial global failure. For example, Gemma-3-27B recovers all evaluated isolated item facts correctly but achieves only 19.75% global validity on eight-item visual instances. Probes decode information about solution membership from global-prompt states, while corresponding clean-state replacement recovers failed decisions over model-dependent depth ranges. These findings show that recovering facts and representing solution-related information are insufficient for reliable joint decisions. They motivate evaluations and model improvements that explicitly target how available evidence is combined, alongside advances in perception and model scale.
| Comments: | 53 pages |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33764 [cs.AI] |
| (or arXiv:2610.33764v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33764 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuyang Dai [view email]
[v1]
Sat, 24 Oct 2026 16:14:52 UTC (2,330 KB)
来源：arXiv cs.AI · arxiv.org