跳到正文
arXiv cs.CL· JunYeong Park, Shijing Wang, Seonwook Park, Michael Stengel, Shalini De Mello, Sang-Il Choi, Hyung Jin Chang·· 23 小时前AI 评分20

Nunchi:用对话线索辅助视觉,提升多人对话中的注视目标识别

Nunchi: Ears Help Eyes in Multi-Party Conversations

AI 导读

研究团队提出 Nunchi 框架,通过将转录文本与注视、说话人模型输出构造成文本提示,让视觉语言模型权衡注视几何与对话线索来识别被注视者,在 SORBET 基准上比最强现有方法高 4.8 个百分点。

正文 · 原文

View PDF HTML (experimental)

Abstract:Identifying whom a person looks at is central to understanding multi-party conversations, yet gaze geometry alone becomes ambiguous as more people appear. Conversations offer cues that geometry lacks, such as who is speaking and what is said, but no gaze dataset provides them for many multi-party scenes. We introduce SORBET, a benchmark of 3,340 samples from 1,578 clips with synchronized speech, transcripts, and attended person labels, covering more multi-party scenes than existing gaze datasets. Prior gaze models struggle to use these cues, which come in different forms, must be linked to the right person, and can conflict with gaze geometry. We propose Nunchi, a framework that constructs a text prompt from the transcript and the outputs of gaze and speaker models, links every cue to the candidates with shared labels, and lets a vision-language model (VLM) weigh gaze geometry against conversational cues to identify the attended person. Nunchi outperforms the strongest existing method by 4.8 percentage points. On SORBET, conversational cues help whereas additional visual cues do not, and the gains hold across three VLMs, with automatic transcripts, and on two existing datasets.
Comments: 10 pages, 5 figures
Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
Cite as: arXiv:2611.23115 [cs.CV]
  (or arXiv:2611.23115v1 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2611.23115

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jun-Yeong Park [view email]
[v1] Tue, 17 Nov 2026 17:35:07 UTC (5,790 KB)

来源:arXiv cs.CL · arxiv.org