arXiv cs.AI· Yulun Jiang, Na\"el Ouerghemi, Yekun Chai, Maria Brbi\'c, Michael Moor·· 10 小时前AI 评分35
MARBLE:面向空间推理与视觉抽象的智能体基准
MARBLE: An Agentic Benchmark for Spatial Reasoning and Visual Abstraction
AI 导读
多模态语言模型基准 MARBLE 发布,覆盖 CUBE、MAZE、BLOX 三个智能体环境,考察 2D/3D 空间谜题中的视觉到空间状态抽象与长程规划。最强模型 GPT-5.4 在最难设置下三个环境平均成功率仅 7.3%。成对消融显示视觉到空间抽象是主要瓶颈:需从图像推断空间状态时,Gemma-4-31B 成功率从 50.7% 降至 3.7%,Kimi-K2.6 从 87.9% 降至 6.5%。
正文 · 原文
Abstract:A central challenge for multimodal language models (MLLMs) in real-world tasks is to abstract visual observations into structured spatial states and reason over how those states change under actions over long horizons. Existing multimodal benchmarks often focus on static question answering, where answers can be directly extracted from images, or assume symbolic states are available and bypass the visual-to-spatial abstraction process. We present MARBLE, a diagnostic benchmark for spatially grounded multimodal reasoning and planning across three agent environments: CUBE, MAZE, and BLOX. MARBLE spans 2D and 3D spatial puzzles under geometric constraints, dynamic layouts, and spatial transformations, with difficulty systematically controlled along two axes: observation format and planning horizon. Extensive evaluations show that frontier MLLMs struggle on MARBLE, with the strongest model GPT-5.4 achieving only a 7.3% average success rate across the three environments in the hardest setting. Pairwise ablations on the open-weight models identify visual-to-spatial abstraction as the major bottleneck: when models must infer spatial states from images instead of receiving human-annotated states, success drops from 50.7% to 3.7% for Gemma-4-31B and from 87.9% to 6.5% for Kimi-K2.6. The results indicate that models often fail to construct precise spatial states from visual observations before planning, a task that is straightforward for humans. Overall, MARBLE reveals the gap in reliable visual-to-spatial abstraction for frontier MLLMs and provides a controlled testbed for measuring progress in spatially grounded multimodal reasoning and planning.
| Comments: | Accepted at NeurIPS 2026, E&D Track(in press) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2506.22992 [cs.AI] |
| (or arXiv:2506.22992v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2506.22992 arXiv-issued DOI via DataCite |
Submission history
From: Michael Moor [view email]
[v1]
Sat, 28 Jun 2025 19:44:32 UTC (5,879 KB)
[v2]
Wed, 18 Nov 2026 17:42:16 UTC (7,646 KB)
来源:arXiv cs.AI · arxiv.org