arXiv cs.AI· Huyu Wu, Meng Tang, Xinhan Zheng, Haiyun Jiang·· 1 天前AI 评分40
语言压过其他模态：多模态大语言模型中的文本主导现象研究
When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models
AI 导读
研究首次系统考察了多模态大语言模型在图像、视频、音频、时间序列和图数据上的文本主导问题，提出 MDI 与 AEI 两项指标量化模态失衡，发现文本主导在所有测试模态中普遍存在。作者归因于非文本模态的 token 冗余导致注意力稀释、融合架构设计以及隐含偏好文本输入的任务形式，并提出一种简单的 token 压缩方法。在 LLaVA-7B 上应用后，其 MDI 从 10.23 降至 0.86。
正文 · 原文
Abstract:Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a diverse range of multimodal tasks. However, these models suffer from a core problem known as text dominance: they depend heavily on text for their inference, while underutilizing other modalities. While prior work has acknowledged this phenomenon in vision-language tasks, often attributing it to data biases or model architectures. In this paper, we conduct the first systematic investigation of text dominance across diverse data modalities, including images, videos, audio, time-series, and graphs. To measure this imbalance, we propose two evaluation metrics: the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI). Our comprehensive analysis reveals that text dominance is both significant and pervasive across all tested modalities. Our in-depth analysis identifies three underlying causes: attention dilution from severe token redundancy in non-textual modalities, the influence of fusion architecture design, and task formulations that implicitly favor textual inputs. Furthermore, we propose a simple token compression method that effectively rebalances model attention. Applying this method to LLaVA-7B, for instance, drastically reduces its MDI from 10.23 to a well-balanced value of 0.86. Our analysis and methodological framework offer a foundation for the development of more equitable and comprehensive multimodal language models.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2508.10552 [cs.CL] |
| (or arXiv:2508.10552v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2508.10552 arXiv-issued DOI via DataCite |
Submission history
From: Meng Tang [view email]
[v1]
Thu, 14 Aug 2025 11:44:52 UTC (4,501 KB)
[v2]
Wed, 11 Nov 2026 22:45:49 UTC (4,581 KB)
来源：arXiv cs.AI · arxiv.org