arXiv cs.CL· Jiaxuan Guo, Xingyu Ren, Kejia Zhang, Jingxin Yang, Youran Sun, Haizhao Yang·· 13 小时前AI 评分34
先问“导演能看到什么”会改变视觉语言模型的选择
Asking What a Director Sees Can Change What Vision-Language Models Choose
AI 导读
在渲染版 Director Task 中，先问模型“导演能看到哪些积木”会使选择被遮挡积木的比例在六个 GPT 模型上上升 12 至 88 个百分点、在 Qwen3.8-27B 上上升 17 个百分点，尽管模型大多能正确报告导演所见。
正文 · 原文
Abstract:Vision-language models are now routinely tested on theory of mind: whether they can work out what another person sees, knows, and means. A common design asks the model to report that perspective and then to act for the person, and reads a correct report beside a wrong action as a failure to use the perspective. That reading rests on a quiet assumption: that asking for the report leaves the action unchanged. We show that it does not. In a rendered Director Task, a partition hides the largest block from a director who asks for "the largest block". Asking first which blocks the director can see raises the share of replies that choose this hidden block by 12 to 88 percentage points on six GPT models and by 17 on Qwen3.8-27B, even though the models mostly report correctly which blocks the director sees. Reasoning traces and prompt rewordings suggest why: the question shifts how the request is read, from the director's view toward the model's own. When the prompt states what the director knows and refers to, fewer than 1% of replies pair a correct report with a hidden-block choice. Any evaluation that elicits a perspective report before the action it scores needs a baseline without the question, and we deliver one: paired prompts that differ only in that question, a design any perspective-taking benchmark can adopt.
| Subjects: | Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.29318 [cs.CL] |
| (or arXiv:2610.29318v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29318 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiaxuan Guo [view email]
[v1]
Wed, 21 Oct 2026 07:21:08 UTC (6,705 KB)
来源：arXiv cs.CL · arxiv.org