arXiv cs.CL· Vansh Bhatia, Mehdi Rezagholizadeh, Parsa Ashrafi Fashi, Igor Gilitschenski, Vikram Appia, Emad Barsoum·· 13 小时前AI 评分38
PRISM-BENCH：诊断超越最终答案准确率的多模态推理
PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy
AI 导读
PRISM-BENCH 是一个包含 1,000 道专家精选问题的基准，覆盖五类推理任务，每题配有经人工标注与验证的理想推理轨迹。感知探针显示约 85% 的前沿模型失败源于感知错误，瓶颈在视觉 grounding；轨迹对齐可预测答案正确性（macro AUROC 0.66）。
正文 · 原文
Abstract:Final-answer accuracy reports whether a vision-language model was right, but not whether its answer was grounded in the image, whether its stated reasoning reflects that evidence, or what the answer cost to produce. Those are the properties that decide whether a system can be deployed. We present PRISM-BENCH, a benchmark of 1,000 expert-curated questions spanning five under-represented reasoning categories, in which every item carries an idealized reasoning trace written by one human annotator and validated by another. The traces support two diagnostics that accuracy cannot provide. A perception probe, in which the model answers from a description it wrote itself, attributes roughly 85% of frontier-model failures to misperception and places the bottleneck at grounding. Trace alignment predicts answer correctness for all 25 systems tested (macro AUROC 0.66) and survives length control, so it measures reasoning content and not verbosity. We further introduce CORE, an efficiency-aware metric that reorders 24.2% of model pairs relative to accuracy while remaining uncorrelated with response length. Across 25 open and closed systems, including GPT-5 and Claude Opus 4.6, the best accuracy is 66.0%. On the four categories with a human baseline the best system trails humans by 18.4 points, and recent frontier gains fall on recognition and not on spatial grounding.
| Comments: | Accepted as a poster at the NeurIPS 2026 Workshop on Grounded and Faithful Vision-Language Models for Real-World Deployment (VLM4RWD). 18 pages, 10 figures, 7 tables. Dataset: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.36554 [cs.CV] |
| (or arXiv:2610.36554v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36554 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vansh Bhatia [view email]
[v1]
Tue, 27 Oct 2026 03:02:39 UTC (6,480 KB)
来源：arXiv cs.CL · arxiv.org