arXiv cs.CL· Haolong Zheng, Maike Z\"ufle, Dominik Mach\'a\v{c}ek, Peter Pol\'ak, Xulin Fan, Xavier Sumba, Siyin Wang, Ond\v{r}ej Klejch, Mark Hasegawa-Johnson·· 9 天前AI 评分30
SteerablePlex：全双工语音模型能否被有效引导？
SteerablePlex: Can We Steer Full-Duplex Models?
AI 导读
研究提出 SimIF-Bench 基准，评估对话模型能否在既定场景内按顺序完成多项目标，结果显示当前开源全双工模型难以遵循此类约束。团队进一步提出基于 GDPO 的训练方案，使全双工模型在保持轮次切换能力的同时遵循对话中的文本指令，所得模型 SteerablePlex 接入异步后端语言模型后，构建出比现有开源模型和 GPT-Realtime 更可靠遵循多阶段约束的可控用户模拟器。
正文 · 原文
Abstract:Full-duplex speech models can listen and speak simultaneously, enabling natural interaction, but become increasingly difficult to control as the conversation history grows. When used as user simulators, this lack of control can cause them to deviate from prescribed scenarios and produce unreliable evaluation outcomes. We introduce SimIF-Bench (Simulator Instruction-Following Benchmark), which evaluates whether a conversational model stays within a prescribed scenario and completes multiple goals in the required order. The benchmark reveals that current open-source full-duplex models struggle to follow such constraints. We then introduce a Group Reward-Decoupled Normalization Policy Optimization (GDPO)-based training recipe that enables a full-duplex model to follow textual instructions during an ongoing conversation while maintaining its turn-taking ability. By connecting the resulting SteerablePlex to an asynchronous backend language model that monitors the conversation and provides instructions when needed, we build a more controllable full-duplex user simulator that follows multi-stage constraints more reliably than existing open-source models and GPT-Realtime.
| Comments: | 5 pages, 2 figures |
| Subjects: | Audio and Speech Processing (eess.AS); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.12201 [eess.AS] |
| (or arXiv:2610.12201v1 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12201 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haolong Zheng [view email]
[v1]
Thu, 8 Oct 2026 15:56:28 UTC (149 KB)
来源：arXiv cs.CL · arxiv.org