arXiv cs.AI· Xixiang He, Baiqi Wu, Xingming Li, Ao Cheng, Qiyao Sun, Xuanyu Ji, Qingyong Hu·· 12 小时前AI 评分40
StemBind：MLLM 在抽象视觉推理中为何“知道规则却选错答案”
StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning
AI 导读
研究者提出诊断基准 StemBind，用同一视觉题干拆成感知、规则、完整作答三个对齐问题，包含 2,298 个题干、19,533 个任务，覆盖九种可审计视觉操作。
正文 · 原文
Abstract:Multimodal large language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We introduce StemBind, a shared-stem diagnostic benchmark that probes the same visual stem with three aligned questions: Perception (what is in the image), Rule (what pattern governs it), and Full (which option completes it), so a final-answer error can be attributed to a specific sub-step on the same evidence. StemBind contains 2,298 curated knowledge-light stems across nine auditable visual operations, totaling 19,533 P/R/F tasks, with each full item annotated by Sternberg's four reasoning stages (S1 Encode, S2 Infer, S3 Map, S4 Apply). Evaluating 24 frontier MLLM configurations yields four findings. (i) The R-F chasm: rule accuracy exceeds full-item accuracy on 22 of 24 models, so most failures happen after the rule is identified. (ii) A persistent binding gap: even when P and R are both correct on the same stem, models still answer F incorrectly 51.2% of the time. (iii) The bottleneck is S3: process diagnostics and Stage-wise Stimulus Augmentation localize the dominant failure to rule-to-instance mapping. (iv) Scaling and thinking do not help: neither larger models nor explicit thinking mode reliably closes the gap, and thinking even lowers rule and full-item accuracy. StemBind reframes AVR evaluation from final-answer ranking to locating where abstract visual reasoning breaks down, identifying rule-to-instance binding as a concrete next target for vision-grounded reasoning.
| Comments: | Accepted at NeurIPS 2026 (Evaluations and Datasets Track). Camera-ready version. Project page: this https URL ; Code: this https URL ; Data: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.00148 [cs.CV] |
| (or arXiv:2606.00148v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.00148 arXiv-issued DOI via DataCite |
Submission history
From: Xixiang He [view email]
[v1]
Fri, 29 May 2026 03:20:05 UTC (8,274 KB)
[v2]
Fri, 23 Oct 2026 13:58:35 UTC (7,776 KB)
来源：arXiv cs.AI · arxiv.org