arXiv cs.AI· Zonghuan Li, Hayley Hung, Chirag Raman·· 3 天前AI 评分25
GesBench：多模态语言模型何时何地感知社会含义？
GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?
AI 导读
研究者提出 GesBench，一个基于人类交互的线索级社会感知基准，用累积子话语片段测量多模态 LLM 何时形成对说话人意图和社会可供性的稳定解读，并用人类注视引导的破坏评估人类关注区域对模型判断是否功能重要。该基准由 6000 多个话语级片段构建，涵盖开放式与闭源多模态模型，发现明显的人机差距：人类更早形成预测性理解且修正更频繁，模型则常在不同片段间含糊或重复。代码已公开。
正文 · 原文
Abstract:Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to specific facial, gestural, or prosodic signals that shape social meaning. Current multimodal LLM benchmarks largely evaluate final-answer understanding over complete episodes, rewarding post-hoc recognition of social meaning rather than the unfolding perception required for real-time interaction. Drawing on interactionally embedded Gestalt principles of multimodal human communication, we introduce GesBench: a human-grounded benchmark for cue-level social perception in multimodal LLMs. GesBench uses cumulative sub-utterance clips to measure when models form stable interpretations of speaker intention and social affordance, and human gaze-guided corruption to evaluate whether regions attended by humans are functionally important for model judgments. Built from more than 6000 utterance-level clips from scripted and unscripted conversational datasets, GesBench includes human annotations of unfolding judgments and gaze on a representative subset. Across open and closed multimodal models, we find a clear human-model gap: humans form predictive understandings early and revise more often, while models often hedge or repeat across different clips. GesBench shifts social evaluation from whether models eventually recognize meaning to when and where that meaning is formed. Our codes are publicly available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15129 [cs.AI] |
| (or arXiv:2610.15129v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15129 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zonghuan Li [view email]
[v1]
Sat, 10 Oct 2026 19:15:11 UTC (7,086 KB)
来源：arXiv cs.AI · arxiv.org