arXiv cs.CL· Oluwanifemi Bamgbose, Jash Shah, Simon Rosen, Lindsay Devon Brin, Hoang H Nguyen, Anke Koelzer, Rachel Hansen, Tara Bogavelli, Fanny Riols·· 13 小时前AI 评分29
研究：MOS 预测器与 Audio-LLM 评判器均无法全面捕捉 TTS 语音的多维感知错误
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
AI 导读
研究将 TTS 的“自然度”拆解为 10 个语言学感知维度，构建首个维度级 TTS 元评测基准，含 860 条由语言学家标注的语音。对 4 个 MOS 预测器和 4 个 Audio-LLM 评判器的评测显示，MOS 预测器更敏感于韵律和副语言层面的声学信号特征，Audio-LLM 评判器则依赖提示词、偏向词级错误且无法泛化到所有维度。
正文 · 原文
Abstract:Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors are more sensitive to acoustic signal-level properties at the prosodic and paralinguistic levels, while Audio-LLM judges show selective, prompt-dependent detection that favors word-level errors and does not generalize across all dimensions. Neither class reliably captures a breadth of linguistically structured, human-perceptible speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.
| Comments: | Accepted to EMNLP 2026 (Main Conference) |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.09930 [cs.SD] |
| (or arXiv:2608.09930v2 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2608.09930 arXiv-issued DOI via DataCite |
Submission history
From: Hoang Nguyen [view email]
[v1]
Mon, 10 Aug 2026 17:59:51 UTC (1,615 KB)
[v2]
Thu, 22 Oct 2026 17:31:03 UTC (251 KB)
来源：arXiv cs.CL · arxiv.org