Uni-Stream：统一流式语音语言模型，同时支持语音合成与识别

Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition

AI 导读

Uni-Stream 是一个基于紧凑预训练语音模型构建的统一流式语音语言模型，以流式方式处理交错的文本与语音，通过两阶段训练达到与任务专用模型相当的性能。研究团队还构建了面向流式训练的数据标注流水线，将公开生成的转写文本和词级时间戳。分析发现，即便在同一模型架构中训练，语音与文本仍处于不同的潜在特征空间，且各模态内部表征在较深层变得更依赖具体任务。

正文 · 原文

View PDF HTML (experimental)

Abstract:Unified streaming speech language models aim to support speech recognition and synthesis within a single model, enabling real-time human-computer interaction. However, jointly supporting these two directions of speech-text conversion while preserving task-specific performance remains challenging. We propose Uni-Stream, a unified streaming speech language model built on compact pretrained speech models. Uni-Stream processes interleaved text and speech in a streaming format and achieves performance comparable to task-specific models through two-stage training. We also develop a data annotation pipeline for streaming training and will publicly release the generated transcriptions and word-level timestamps. We further examine Uni-Stream's representations across tasks and modalities and find that speech and text residence in distinct latent feature space even when trained in a unified model architecture, while representations within each modality become more task-dependent in later layers.

Subjects:	Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.32187 [eess.AS]
	(or arXiv:2610.32187v1 [eess.AS] for this version)
	https://doi.org/10.48550/arXiv.2610.32187 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xuan Shi [view email]
[v1] Fri, 23 Oct 2026 05:32:28 UTC (224 KB)

来源：arXiv cs.AI · arxiv.org