arXiv cs.AI· Annika Michael, Zachary Ellis, Yuxiang Zhou, Samuel Thio, Yajie Vera He, Spencer Hazel, Jared Joselowitz·· 4 小时前AI 评分31
语音到底重不重要?在已部署临床语音智能体通话上评估患者挫败感检测
Does the Voice Matter? Evaluating Frustration Detection on Deployed Clinical Voice-Agent Calls
AI 导读
针对已部署临床语音智能体的真实患者通话,研究发现仅靠语音转写文本无法在单词话轮上排除患者挫败感——在 30 通电话的 39 个挫败性单词话轮中,纯文本只识别出 3 或 4 个,仅凭词数的分类器准确率达 57.3%。
正文 · 原文
Abstract:Autonomous voice agents now conduct routine clinical follow-up calls, so some automated triage is unavoidable. A patient becoming frustrated is one thing that triage should catch. These systems typically read only the transcript the speech-to-text stage produces, not the voice, and whether that is enough is largely untested on real clinical speech. We study per-turn frustration detection on real patient calls from a deployed clinical voice agent: 150 turns from six calls labelled by three annotators, with a length-matched acted corpus as a control, and 714 turns from 30 calls. Patient turns are overwhelmingly short: across the deployed calls, roughly 59% are one or two words, against 13% in the full acted corpus, and a corpus accuracy gap remains even after the acted set is length-matched. On one-word turns, transcript-only detection does not exclude chance. Across five audio-capable language models, the transcript flags 3 or 4 of the 39 frustrated one-word turns in the 30 calls, and a classifier given only the word count reaches 57.3%. Whether audio helps depends on the model and on the instruction. Asked to attend to delivery, two Gemini 2.5 models gain 3.0 pp over the transcript across 30 calls (-1.0 to 7.5 pp). Of the turns they confidently clear, 6.9% were in fact frustrated for audio against 13.0% for the transcript, a difference of 6.1 pp (2.2 to 10.4 pp). Across the 30 calls, given one prompt identical for both inputs, no model's audio-text interval excludes zero. Under the input-specific prompts, a newer Gemini model and two smaller open models do worse with audio than with the transcript (-3.5, -8.6 and -10.6 pp). A WavLM probe trained on the clinical calls matches the best prompted model overall (71.6%) but misses every frustrated one-word turn in the 24 added calls. When a patient's turn is short, the transcript alone is not enough to rule out frustration.
| Comments: | Extended version of papers accepted at ML4H 2026 (Proceedings) and GenAI4H workshop @ NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.21001 [cs.CL] |
| (or arXiv:2611.21001v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21001 arXiv-issued DOI via DataCite |
Submission history
From: Annika Michael Ms [view email]
[v1]
Mon, 16 Nov 2026 12:27:56 UTC (1,879 KB)
[v2]
Tue, 17 Nov 2026 12:06:20 UTC (1,879 KB)
来源:arXiv cs.AI · arxiv.org