arXiv cs.CL· Riqiang Wang, Elena Khasanova, Harsh Saini, Lex Konnelly, Parsa Kavehzadeh, Matthias Lee, Mohamed Attia·· 9 天前AI 评分22
提示词 vs 规则：语音用户模拟器中语音自然度行为的审计与控制
Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators
AI 导读
研究审计了基于 LLM 提示的 tau-Voice 与规则注入算法在语音用户模拟器中实现不流畅、打断和反馈三种自然度行为的效果。结果显示，提示词方式不可靠，生成的语音与指令不一致，位置和分布不如指令所暗示的自然；而基于规则、无需模型的算法能产生更可控、更多样且更贴近自然语音的行为。研究认为，未经用户模拟专门训练的 LLM 不足以单独代表真实用户行为，需要语言学驱动的确定性方法或专用模型来弥补差距。
正文 · 原文
Abstract:As voice agents gain more popularity commercially, the user simulators used to evaluate the deployed agents are also being developed to include more realistic, variable, and diverse speech naturalness behaviors -- disfluency, interruption and backchanneling. The quality of the user simulator directly affects the validity of agent evaluation results. However, we find that most studies so far have not examined in detail whether the intended configuration for these behaviors is realized in the simulation. In this study, we audit the realized naturalness behaviors of tau-Voice, our own LLM-based prompting approach across three models, and our rule-based injection algorithm for disfluency, interruption, and backchanneling. We find that prompting for these behaviors is unreliable and produces speech inconsistent with the instructions, placed and distributed less naturally than the instruction implies. In contrast, our rule-based, model-free algorithm produces controllable and diverse naturalness behaviors more aligned with natural speech. Our results suggest that LLMs not purpose-trained for user simulation are not sufficient on their own to represent authentic user behavior, and that linguistically informed deterministic approaches or specialized models are needed to close the gap; auditing and reporting realized naturalness behaviors, rather than configured settings, is what makes that gap visible.
| Comments: | Accepted to the UserSim @ NeurIPS 2026 workshop (non-archival) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.11015 [cs.CL] |
| (or arXiv:2610.11015v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11015 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Elena Khasanova [view email]
[v1]
Wed, 7 Oct 2026 23:55:27 UTC (26 KB)
来源：arXiv cs.CL · arxiv.org