arXiv cs.CL· Chengfei Wu, Ronald Seoh, Bingxuan Li, Liqiang Zhang, Fengrong Han, Dan Goldwasser·· 2 天前AI 评分29
MagiC：评估多模态认知的视觉推理基准
MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning
AI 导读
研究者推出 MagiC 基准，用于评估大视觉语言模型是否真正进行有依据的视觉推理，而非依赖表面模式与数据集偏差。该基准包含约 5,500 条弱监督 QA 样本和 900 条人工精标样本（含答案、推理依据与边界框标注），在四个维度上评测了 15 个 7B 至 70B 参数的视觉语言模型，并引入 MagiScore 与 StepSense 新指标。
正文 · 原文
Abstract:Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, it remains unclear whether these models genuinely perform grounded visual reasoning or rely on superficial patterns and dataset biases. In this work, we introduce MagiC, a comprehensive benchmark designed to evaluate grounded multimodal cognition, assessing not only answer accuracy but also the quality of step-by-step reasoning and its alignment with relevant visual evidence. Our benchmark includes approximately 5,500 weakly supervised QA examples generated from strong model outputs and 900 human-curated examples with fine-grained annotations, including answers, rationales, and bounding box groundings. We evaluate 15 vision-language models ranging from 7B to 70B parameters across four dimensions: final answer correctness, reasoning validity, grounding fidelity, and self-correction ability. MagiC further includes diagnostic settings to probe model robustness under adversarial visual cues and assess their capacity for introspective error correction. We introduce new metrics such as MagiScore and StepSense, and provide comprehensive analyses that reveal key limitations and opportunities in current approaches to grounded visual reasoning.
| Comments: | AACL 2026 Main Conference |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2507.07297 [cs.CV] |
| (or arXiv:2507.07297v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2507.07297 arXiv-issued DOI via DataCite |
Submission history
From: Chengfei Wu [view email]
[v1]
Wed, 9 Jul 2025 21:44:12 UTC (21,942 KB)
[v2]
Sun, 4 Oct 2026 04:09:29 UTC (23,161 KB)
来源：arXiv cs.CL · arxiv.org