arXiv cs.AI· Shuo Wen, Beixi Du, Edwin Meriaux, Junming Shi, Chloe Si, David Meger, Doina Precup, Gregory Dudek·· 14 小时前AI 评分35
The Prestige：用魔术表演评测 VLM 的认知视觉推理能力
The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks
AI 导读
研究者推出视频理解基准 The Prestige，用专业魔术表演中的误导设计考察 VLM 追踪事件、解释意外结果的能力，题目涵盖时序与因果理解及对抗性问题鲁棒性。评测显示最强开源与闭源 VLM 均与人类表现存在明显差距，且人类与自动评分之间存在系统性偏差。消融实验还发现，模型在无法独立生成正确解释时仍能选对多选题选项。
正文 · 原文
Abstract:We introduce The Prestige, a video understanding benchmark that probes cognitive visual reasoning through professional magic performances, where deliberate misdirection challenges models to track events and explain unexpected outcomes. The benchmark comprises open-ended questions ranging from temporal and causal understanding to robustness against adversarial questions. Evaluation of leading open-source and proprietary VLMs reveals a substantial gap between even the strongest model and human performance. We also find systematic discrepancies between human and automated grading and identify recurring model failure modes. Ablations further show that models can select correct multiple-choice answers even when they fail to generate correct explanations independently, motivating further investigations into question formats. Together, these findings highlight limitations in both VLM reasoning and its evaluation, underscoring the need for more reliable video understanding models and more rigorous benchmarks.
| Comments: | 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Track on Evaluations and Datasets |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38103 [cs.CV] |
| (or arXiv:2610.38103v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38103 arXiv-issued DOI via DataCite |
Submission history
From: Shuo Wen [view email]
[v1]
Wed, 28 Oct 2026 00:41:06 UTC (8,484 KB)
来源：arXiv cs.AI · arxiv.org