arXiv cs.AI· Zixin Su, Yangguang Lin, Zegang Peng, Yuntong Sun, Xuanyi Wang, Jiahao Wang, Qiankai Xu, Bing Yuan, Jingjia Huang, Ruyang Liu, Yanxu Hu, Yufeng Zhou, Yi Lin·· 1 天前AI 评分40
HumanSightBench：评估 MLLM 人类视觉感知能力的基准
HumanSightBench: A Benchmark for Human Visual Perception in MLLMs
AI 导读
研究者提出 HumanSightBench，一个包含 2,876 道短答案视觉问题、覆盖 7 个顶层感知类别和 21 个子类别的基准，用于评估 MLLM 与人类视觉感知的差距。所有题目需通过问题清晰度、答案唯一性、视觉 grounding、人类可解性和纯文本捷径过滤五道纳入门槛。在八个受评 MLLM 中最佳 pass@3 仅为 65.6%，Counting 类别无一模型超过 27.5%。
正文 · 原文
Authors:Zixin Su, Yangguang Lin, Zegang Peng, Yuntong Sun, Xuanyi Wang, Jiahao Wang, Qiankai Xu, Bing Yuan, Jingjia Huang, Ruyang Liu, Yanxu Hu, Yufeng Zhou, Yi Lin
Abstract:Multimodal large language models (MLLMs) have demonstrated strong performance on knowledge-intensive and reasoning-based vision-language benchmark datasets, but still struggle to solve the visual questions that people can solve easily. Evaluating the differences between MLLMs and human-level visual perception is challenging due to end-to-end evaluation which combines various skills, or because we are uncertain if individual items are valid. Specifically, observed errors may reflect visual perception, language priors, background knowledge, question ambiguity, and underspecified answers. Our guideline for HumanSightBench is that we construct items that are human-easy by construction and model-hard by evaluation. HumanSightBench contains 2,876 visual questions with short answers across 7 top-level perceptual categories and 21 subcategories. We only include items that pass five inclusion gates: question clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcut filtering. According to the protocol, errors are defined as incorrect answers on human-validated visual tasks with the evaluated interface. The inclusion gates are intended to reduce confounds from ambiguity and external knowledge. The best pass@3 score is 65.6% across eight evaluated MLLMs. No evaluated model exceeds 27.5% on Counting. These results identify capability gaps that are hidden by overall scores and can help future training experiments and held-out evaluation.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.12305 [cs.CV] |
| (or arXiv:2611.12305v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.12305 arXiv-issued DOI via DataCite |
Submission history
From: Zixin Su [view email]
[v1]
Mon, 9 Nov 2026 12:27:19 UTC (1,554 KB)
[v2]
Wed, 11 Nov 2026 10:19:36 UTC (1,553 KB)
来源：arXiv cs.AI · arxiv.org