arXiv cs.CL· Maya K. Nachesa, Vlad Niculae, Vagrant Gautam·· 11 小时前AI 评分42
多模态语音识别模型存在人口统计学偏见：mWhisper-Flamingo 与 Gemini 词错率最高相差 4.05 点
Your Multimodal Speech Model Says I Have a Face for Radio
AI 导读
研究首次对多模态语音识别开展偏见评估，用唇同步技术将不同人脸与同一段音频配对生成视频，测量转写准确率变化。在 mWhisper-Flamingo 和 Gemini 上，按自报性别、族裔及其交叉分组，词错率最多下降 4.05 点。作者指出，增加模态信号未必更好，开发者应评估、修复并公开此类局限。
正文 · 原文
Abstract:As large neural models have become better at language tasks, researchers are increasingly building multi- and omnimodal models that handle more than one modality of data. One example is the expansion of speech recognition models to audio-visual data, the goal being noise mitigation and multimodal subtitling. While performance and bias have been studied extensively in the single-modality regime, it is unknown how new modalities affect this, even though they produce biases in humans. We therefore propose the first bias evaluation of multimodal speech recognition, where we create videos pairing different faces with the same audio using lip-sync, and measure changes in speech transcription accuracy. We find large quality-of-service differences across mWhisper-Flamingo and Gemini models, with drops of up to 4.05 word error rate points, across self-declared gender, ethnicity, and their intersection. Our findings point to a priority for developers to evaluate, fix, and communicate such limitations, as providing more signals through additional modalities is not necessarily better, and may even lead to biased outcomes.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.30472 [cs.CL] |
| (or arXiv:2605.30472v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.30472 arXiv-issued DOI via DataCite |
Submission history
From: Maya K. Nachesa [view email]
[v1]
Thu, 28 May 2026 18:42:38 UTC (78 KB)
[v2]
Thu, 5 Nov 2026 14:39:50 UTC (96 KB)
来源：arXiv cs.CL · arxiv.org