DEAF：音频语言模型声学忠实度诊断评测基准

View PDF HTML (experimental)

Abstract:Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness), a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven Audio MLLMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding.

Comments:	14 pages,6 figures
Subjects:	Artificial Intelligence (cs.AI); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Cite as:	arXiv:2603.18048 [cs.AI]
	(or arXiv:2603.18048v3 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2603.18048 arXiv-issued DOI via DataCite

Submission history

From: Jiaqi Xiong [view email]
[v1] Tue, 17 Mar 2026 15:52:26 UTC (12,711 KB)
[v2] Fri, 20 Mar 2026 01:17:57 UTC (12,711 KB)
[v3] Fri, 9 Oct 2026 17:56:35 UTC (16,487 KB)