arXiv cs.CL· Wish Suharitdamrong, Muhammad Awais, Xiatian Zhu, Sara Atito·· 11 小时前AI 评分34
从感知到决策：多模态 LLM 中听觉与视觉信息流的内部路径研究
From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
AI 导读
研究追踪了音视频大语言模型（AVLLMs）内部音频与视觉 token 的信息流动路径：在音视频视频输入下，模型沿用 VLM 与 VideoLLM 的序列信息流路径，音频与视觉的贡献按任务对各模态的依赖比例分配；在多段交错音视频条目场景下，路由则转为不同的并行流。
正文 · 原文
Abstract:Multimodal Large Language Models (MLLMs) can listen and see, but how do audio and visual signals actually travel through the network to shape an answer? Despite their growing role in research and real-world applications, the internal pathways through which audio and visual tokens influence the final prediction remain poorly understood. In this study, we examine audio-visual information flow inside Audio-Visual Large Language Models (AVLLMs), tracing how AVLLMs route, utilize, and integrate audio and visual information across two input configurations, audio-visual video and multiple interleaved audio-visual items. We find that for audio-visual video, AVLLMs follow the sequential information flow pathway established for VLMs and VideoLLMs, with audio and visual contribution flowing along this pathway in proportion to the task's reliance on each modality. In settings with multiple interleaved audio-visual items, this routing shifts to different parallel streams. Furthermore, we demonstrate that audio-visual and other token types can be discarded once their information is transferred to LLM, with minimal impact on the model's prediction or even slight improvement, generalizing across multiple tasks and datasets, enabling more efficient inference. These findings hold across multiple models and scales, Qwen2.5-Omni and Video-SALMONN2 Plus at 3B and 7B scales, leading to hypotheses on why these flow structures emerge. Together, these results deliver the first coherent picture of how AVLLMs orchestrate sound and sight inside the network and lay the groundwork for the next wave of interpretability, design, and efficiency advances in audio-visual and broader MLLMs.
| Comments: | Accepted by NeurIPS 2026, 54 pages, 34 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD) |
| Cite as: | arXiv:2606.10147 [cs.AI] |
| (or arXiv:2606.10147v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.10147 arXiv-issued DOI via DataCite |
Submission history
From: Wish Suharitdamrong [view email]
[v1]
Mon, 8 Jun 2026 20:26:09 UTC (3,121 KB)
[v2]
Mon, 2 Nov 2026 21:15:54 UTC (3,369 KB)
[v3]
Thu, 5 Nov 2026 20:23:53 UTC (3,369 KB)
来源：arXiv cs.CL · arxiv.org