arXiv cs.AI· Yiran Huang, Zeyong Zhang, Bharat Biswal, Xin Di, Mengjia Xu·· 12 小时前AI 评分29
面向卡通电影社交与情感理解的大脑对齐多模态 LLM
Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies
AI 导读
研究对比了扩散模型与自回归多模态 LLM 在三部卡通短片上的表现，用连续人类心理理论（ToM）与情感效价评分检验其时间对齐度，并将各来源评分分别用于受试者回归模型预测脑区 fMRI 响应。
正文 · 原文
Abstract:Multimodal large language models (MLLMs) offer scalable annotations of social and affective content, but their temporal agreement with human judgments and neural predictive value require further evaluation. We compared diffusion-based and autoregressive MLLMs with continuous human Theory of Mind (ToM) and affective valence ratings across three short cartoon movies. Behavioral analyses assessed temporally aligned Pearson correlations, rating distributions, and the effects of preceding scene descriptions. Ratings from each source were separately used in subject-specific regression models to predict regional fMRI responses. Model ratings captured aspects of human temporal variation despite systematic distributional differences, with generally stronger alignment for valence than ToM. Preceding context improved alignment in some conditions, depending on the model and movie. In \emph{The Present}, more regions showed significant encoding for MLLM ratings than for human ratings, whereas effects were more limited in the other movies. Spatial overlap across rating sources occurred mainly among MLLMs; overlap between human and model ratings was limited and involved primarily visual, dorsal-attention, and posterior default-mode regions. These findings support MLLMs as scalable tools for generating brain-predictive annotations of social and emotional content, while highlighting systematic rating differences and model- and stimulus-dependent alignment with human judgments.
| Subjects: | Artificial Intelligence (cs.AI); Neurons and Cognition (q-bio.NC) |
| Cite as: | arXiv:2610.22706 [cs.AI] |
| (or arXiv:2610.22706v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22706 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yiran Huang [view email]
[v1]
Thu, 15 Oct 2026 22:22:03 UTC (10,932 KB)
来源：arXiv cs.AI · arxiv.org