arXiv cs.CL· Yin Liu·· 11 小时前AI 评分47
研究揭示多模态 LLM 在图表计数上的失败根源：是看错还是算错
Seeing vs. Counting: Complexity Twins Show Where Multimodal LLMs Fail on Diagrams
AI 导读
研究用同一张图的不同计数复杂度问题（生成树 vs 简单路径）区分多模态 LLM 的"看"与"算"，测试 Gemini、Gemma 和 Qwen3-VL。Gemini 模型在思维链下多数能正确读取小图却算错，答对几率随答案规模每增大一个数量级下降 2.38 倍，而非由复杂度类别决定。
正文 · 原文
Abstract:When a multimodal LLM miscounts in a diagram, does it fail at seeing or at counting? Existing visual graph and puzzle benchmarks vary visual density and algorithmic hardness together, confounding the two. We pair counting questions about the same drawing that sit in different counting-complexity classes, such as its spanning trees (a determinant) and its simple paths (#P-complete), and use a transcription probe and a text-only control to separate reading from computing. We test Gemini, Gemma and Qwen3-VL models; for the Gemini models, the answer depends on how the model computes. In chain of thought, most of them read small drawings correctly and then fail at counting, at a rate set far more by the size of the answer than by its complexity class (for the primary model, the odds of an exact answer fall 2.38-fold per decade of the answer). Given code and an exact transcription, the primary model counts correctly even on 10x10 grids, and reading becomes the bottleneck: a single view misses horizontal edges, and a second reading of a rotated copy with crop checks reads 86/96 large grids exactly, against 57/96 when the upright reading is simply repeated.
| Comments: | 8 pages main text + 96 pages appendix (107 pages in total, including references) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.08998 [cs.CV] |
| (or arXiv:2611.08998v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08998 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yin Liu [view email]
[v1]
Fri, 6 Nov 2026 05:17:08 UTC (1,761 KB)
来源：arXiv cs.CL · arxiv.org