arXiv cs.CL· Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo, Quinten McNamara, Jonas Robertson, Shashi Bhushan TN·· 13 小时前AI 评分34
从文本到语音：一个可复现、可验证的工具调用 LLM 智能体评测框架
From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
AI 导读
研究团队提出一个数据集无关的框架，用 TTS、说话人变化和环境噪声把已验证的文本工具调用基准转成配对的文本-音频评测，无需重新标注工具 schema 和标准答案。
正文 · 原文
Abstract:Voice agents increasingly require reliable tool use from speech, whereas prominent tool-calling benchmarks remain text-based. We study whether verified text benchmarks can be converted into controlled audio-based tool calling evaluations without re-annotating the tool schema and gold labels. Our dataset-agnostic framework uses text-to-speech, speaker variation, and environmental noise to create paired text-audio instances while preserving the original dataset annotations. Based on extensive evaluation of 7 omni-modal models on audio-converted versions of Confetti and When2Call, our framework demonstrates that the performance is strongly model- and task-dependent: Gemini-3.1-Flash-Live obtains the highest Confetti score (70.4), whereas GPT-Realtime-1.5 performs best on When2Call (71.9). On Confetti, the text-to-voice gap ranges from 1.8 points for Qwen3-Omni to 4.8 points for GPT-Realtime-1.5. A targeted analysis of failure cases demonstrates that degradations most often reflect misunderstandings of argument values in the speech. Considering real-world deployment scenarios, we further report text-only results, an ambiguity-based reformulation stress test, and a reference-free LLM-as-judge protocol validated against human preferences. Notably, we find that open-source Qwen3 judges with at least 8B parameters exceed 80% agreement with proprietary judges, supporting privacy-preserving evaluation. Overall, our framework provides a verifiable and reproducible first-stage diagnostic that complements purpose-built audio corpora.
| Comments: | Accepted to Transactions on Machine Learning Research (TMLR). The final version will appear here: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.15104 [cs.CL] |
| (or arXiv:2605.15104v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15104 arXiv-issued DOI via DataCite |
Submission history
From: Md Tahmid Rahman Laskar [view email]
[v1]
Thu, 14 May 2026 17:22:42 UTC (687 KB)
[v2]
Wed, 20 May 2026 10:20:54 UTC (687 KB)
[v3]
Wed, 21 Oct 2026 16:44:29 UTC (687 KB)
来源：arXiv cs.CL · arxiv.org