arXiv cs.CL· Arnav Verma, Kushin Mukherjee, Christopher Potts, Elisa Kreiss, Judith E. Fan·· 13 小时前AI 评分31
以人为中心评估视觉语言模型对数据可视化的理解能力
Human-centered evaluation of data visualization understanding in vision-language models
AI 导读
研究者提出一套以人为中心的评估协议，在相同可视化理解题目上直接对比视觉语言模型与人类的作答模式。该协议应用于 8 个视觉语言模型和 6 项测评（含 5 项人类可视化素养测试和 1 项专为模型设计的测试），模型平均表现低于人类，即使放宽标准差距依然存在。模型与人类在题目相对表现上有所相关，但所有模型的错误模式都与人类可靠地不同。
正文 · 原文
Abstract:Data visualization understanding, which requires integrating visual, numerical, and linguistic inputs, poses fundamental challenges to multimodal reasoning. Recently developed vision-language models (VLMs) are promising candidates for modeling the computations that underlie human data visualization understanding. However, whether they are useful as scientific models depends not only on whether they produce correct answers, but also whether they approximate the patterns of responses that people produce on the same tasks. This distinction has been ignored in recent evaluations of VLMs, which typically only report overall model performance (accuracy) without direct comparison to human behavioral patterns. Here we introduce a human-centered evaluation protocol that compares VLM and human responses on the same set of visualization understanding items. We apply this protocol to eight VLMs across six assessments: five established tests of human visualization literacy, plus one designed specifically for evaluating VLMs. Models performed worse than humans on average, and this gap persisted even under lenient criteria. Moreover, while relative performance across items was somewhat correlated between models and humans, all models produced error patterns that were reliably distinct from those of humans. We envision our evaluation protocol helping to identify artificial systems that might serve as promising scientific cognitive models of how humans reason about data visualizations.
| Subjects: | Human-Computer Interaction (cs.HC); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2505.17202 [cs.HC] |
| (or arXiv:2505.17202v2 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2505.17202 arXiv-issued DOI via DataCite |
|
| Journal reference: | Scientific Reports (2026) |
| Related DOI: | https://doi.org/10.1038/s41598-026-68119-z
DOI(s) linking to related resources |
Submission history
From: Arnav Verma [view email]
[v1]
Thu, 22 May 2025 18:15:04 UTC (9,881 KB)
[v2]
Tue, 20 Oct 2026 18:55:19 UTC (6,301 KB)
来源：arXiv cs.CL · arxiv.org