arXiv cs.CL· Yuan Ge, Pei Zhang, Xize Cheng, Lingzhou Mu, Mingyang Wang, Jin Xu, Baosong Yang, Shengxiang Gao, Jingbo Zhu, Zhengtao Yu, Tong Xiao·· 13 小时前AI 评分34
Omnilingual-MsASR：评测全模态大模型的多语言长时多说话人 ASR 基准
Omnilingual-MsASR: Benchmarking Multimodal, Multilingual, Long-Form Multi-Speaker ASR of Omni Large Language Models
AI 导读
研究者推出 Omnilingual-MsASR，一个面向长时多说话人自动语音识别的多语言、多模态基准，覆盖 6 种语言、最长 30 分钟录音、最多 20 名说话人，并提供平行音频与视频输入以考察跨模态一致性。对领先全模态 LLM 的评测显示明显性能瓶颈，tcpWER 常超过 100%，细粒度时间定位、说话人区分与跨模态协同成为关键挑战。
正文 · 原文
Authors:Yuan Ge, Pei Zhang, Xize Cheng, Lingzhou Mu, Mingyang Wang, Jin Xu, Baosong Yang, Shengxiang Gao, Jingbo Zhu, Zhengtao Yu, Tong Xiao
Abstract:Omni-modal large language models (LLMs) have achieved remarkable progress in audio and video understanding. However, existing models struggle with long-form, multi-speaker videos, especially in understanding the entire video and recognizing who says what and when. To characterize the capability limit of current omni LLMs and guide future model development, we introduce Omnilingual-MsASR, a multilingual, multimodal benchmark for long-form Multi-speaker Automatic Speech Recognition (MsASR) covering six languages, recordings up to 30 minutes, and scenarios with up to 20 speakers. Omnilingual-MsASR provides parallel audio and video inputs to evaluate models' cross-modal consistency and their ability to exploit complementary cues across modalities. Extensive evaluations of leading omni-modal LLMs reveal substantial performance bottlenecks, while their tcpWERs frequently exceed 100%. These findings highlight fine-grained temporal localization, speaker disambiguation, and cross-modal synergy as key challenges for future omni-modal foundation models. Our work is available at this https URL.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.29455 [cs.CL] |
| (or arXiv:2610.29455v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29455 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuan Ge [view email]
[v1]
Wed, 21 Oct 2026 09:11:30 UTC (2,496 KB)
来源：arXiv cs.CL · arxiv.org