arXiv cs.CL· Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal·· 11 小时前AI 评分31
StreamGaze：首个评估 MLLM 在流式视频中利用视线进行时序与主动推理的基准
StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
AI 导读
StreamGaze 是首个评估多模态大语言模型（MLLM）如何在流式视频中利用人类视线信号进行时序与主动推理的基准，涵盖过去、现在与主动预测三类任务。该基准通过注视点提取、区域视觉提示和扫描路径构建，将第一人称视频与原始视线轨迹对齐，生成时空接地的 QA 对。在所有任务上，SOTA MLLM 与人类表现存在显著差距，暴露了视线时序推理、意图建模与主动预测的局限。
正文 · 原文
Abstract:Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intent for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. Past and present tasks require immediate responses based on observations available at query time, whereas proactive tasks require models to continuously process incoming frames and respond at the appropriate future moment. To build StreamGaze, we develop a gaze-video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intent modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support research in gaze-guided streaming video understanding.
| Comments: | Accepted to the 40th Annual Conference on Neural Information Processing Systems (NeurIPS 2026) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2512.01707 [cs.CV] |
| (or arXiv:2512.01707v4 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2512.01707 arXiv-issued DOI via DataCite |
Submission history
From: Daeun Lee [view email]
[v1]
Mon, 1 Dec 2025 14:15:44 UTC (4,693 KB)
[v2]
Fri, 27 Mar 2026 17:30:08 UTC (4,748 KB)
[v3]
Wed, 13 May 2026 04:12:19 UTC (4,748 KB)
[v4]
Thu, 5 Nov 2026 15:52:54 UTC (5,323 KB)
来源：arXiv cs.CL · arxiv.org