跳到正文
arXiv cs.CL· Zehan Li, Hongjie Chen, Qing Wang, Yuxin Zhang, Jing Zhou, Hang Lv, Mengjie Du, Yaodong Song, Jie Lian, Jian Kang, Jie Li, Yongxiang Li·· 10 小时前AI 评分33

TELEVAL:面向中文交互场景的语音语言模型基准测试

TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios

AI 导读

研究者推出 TELEVAL,一个面向中文语音交互的大规模基准,在不提供任务专用提示词或预设选项的情况下评估语音语言模型(SLM)的真实交互表现。该基准从可靠内容完成度与交互适当性两方面评测,实验发现模型在中文文化问题、方言语音等场景下表现不佳,且难以根据听觉线索调整回应,并出现将感知到的听觉线索描述出来而非回应用户需求的“Caption Trap”失败模式。

正文 · 原文

Authors:Zehan Li, Hongjie Chen, Qing Wang, Yuxin Zhang, Jing Zhou, Hang Lv, Mengjie Du, Yaodong Song, Jie Lian, Jian Kang, Jie Li, Yongxiang Li

View PDF HTML (experimental)

Abstract:Spoken Language Models (SLMs) are expected to support natural spoken interaction beyond task completion. Existing benchmarks assess a range of semantic tasks and conversational capabilities. However, high task scores do not necessarily imply reliable and appropriate spoken interaction. To address this gap, we introduce TELEVAL, a large-scale benchmark focused on Chinese spoken interaction. It evaluates responses to spoken inputs without additional task-specific prompts or predefined answer options. TELEVAL covers two complementary aspects: (1) Reliable Content Fulfillment, which measures semantic accuracy under diverse acoustic and linguistic conditions, and (2) Interactional Appropriateness, which assesses conversational naturalness and appropriate adaptation to auditory cues. Experiments reveal limitations in both aspects. Strong results on factual question-answering datasets do not consistently extend to locally relevant knowledge or challenging speech conditions, such as Chinese cultural questions and dialectal speech. Models also struggle to adapt their responses to auditory cues, even in cases where they explicitly recognize them. We further document a recurring failure pattern, termed the "Caption Trap", in which some responses describe a perceived auditory cue instead of addressing the user's conversational needs. Overall, the evaluated SLMs struggle to deliver consistently accurate and conversationally appropriate responses. TELEVAL provides a diagnostic framework for evaluating both content reliability and conversational adaptation.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Cite as: arXiv:2507.18061 [cs.CL]
  (or arXiv:2507.18061v5 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2507.18061

arXiv-issued DOI via DataCite

Submission history

From: Zehan Li [view email]
[v1] Thu, 24 Jul 2025 03:23:55 UTC (356 KB)
[v2] Tue, 6 Jan 2026 08:48:39 UTC (533 KB)
[v3] Mon, 12 Jan 2026 03:01:57 UTC (533 KB)
[v4] Mon, 24 Aug 2026 13:42:31 UTC (542 KB)
[v5] Thu, 19 Nov 2026 11:22:31 UTC (589 KB)

来源:arXiv cs.CL · arxiv.org