arXiv cs.CL· Rahul Murali Shankar, Titus von der Malsburg, Sebastian Pad\'o·· 3 天前AI 评分20
现成语言-视觉编码器可预测视觉世界实验中的注视偏好
Off-the-Shelf Language-Vision Encoders Predict Human Gaze Preferences in Visual World Experiments
AI 导读
一项研究提出用现成的双模态 CLIP 双编码器预测视觉世界实验中的注视偏好，通过对该模型施加归因方法推导目标偏好，无需生成式架构、无需微调，尽管模型并未针对该任务训练。该方法能定性复现一项经典英语视觉世界研究中人类预测性加工的注视模式。
正文 · 原文
Abstract:A central question for computational psycholinguistics is to what extent neural language models can serve as cognitively adequate models of human language processing. However, work has been overwhelmingly focused on the unimodal case of written or spoken language. In contrast, multimodal paradigms, like visual world experiments that present participants with both visual and linguistic input simultaneously, have been neglected. In this paper, we present a novel approach that predicts patterns of gaze preferences in visual world studies. To do so, it derives target preferences by applying an attribution method to a simple bi-modal CLIP bi-encoder. We demonstrate the ability of this approach to qualitatively replicate the gaze patterns found in a seminal English visual world study demonstrating human predictive processing. It does so without a generative architecture and without the need for fine-tuning, despite not being trained for the task.
| Subjects: | Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2608.07282 [cs.CL] |
| (or arXiv:2608.07282v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.07282 arXiv-issued DOI via DataCite |
Submission history
From: Sebastian Pado [view email]
[v1]
Fri, 7 Aug 2026 14:42:51 UTC (752 KB)
[v2]
Wed, 14 Oct 2026 06:32:56 UTC (869 KB)
来源：arXiv cs.CL · arxiv.org