arXiv cs.CL· Hao Zhang, Ismail Rasim Ulgen, Bismarck Bamfo Odoom, Rui Liu, Zihan Zhang, Berrak Sisman, Philipp Koehn·· 11 小时前AI 评分26
面向表现力语音到语音翻译的全方位平行监督
Holistic Parallel Supervision for Expressive Speech to Speech Translation
AI 导读
研究提出一种训练数据构建策略，将专业配音与多语言语音转换结合，为语音到语音翻译（S2ST）提供在内容、韵律、时序和说话人身份上对齐的全方位平行监督。基于该监督训练的端到端方法 DirectS2ST 直接从源语音预测完整目标语音表示，在英语-德语和英语-西班牙语实验中于内容、韵律、时序、说话人相似度和人工评分表现力上均取得最大提升。
正文 · 原文
Abstract:Expressiveness is central to speech-to-speech translation (S2ST), yet existing datasets rarely align source and target speech jointly in content, prosody, timing, and speaker identity. Consequently, current systems often rely on indirect conditioning or transfer mechanisms. To obtain this supervision, we introduce a training-data construction strategy that combines professional dubbing, which provides human-performed correspondence in content, prosody, and timing, with multilingual voice conversion to reduce speaker mismatch. We evaluate this supervision using existing expressive S2ST systems and DirectS2ST, an end-to-end method developed in this work that predicts the complete target-speech representation directly from source speech. Experiments on English-German and English-Spanish show that holistic parallel supervision improves expressive realization across all evaluated systems. DirectS2ST exhibits the largest gains in content, prosody, timing, speaker similarity, and human-rated expressiveness. To our knowledge, this work provides the first systematic investigation of holistic parallel supervision for expressive S2ST.
| Comments: | 5 pages, 1 figure, 5 tables. Submitted to ICASSP 2027 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.06010 [cs.CL] |
| (or arXiv:2611.06010v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06010 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hao Zhang [view email]
[v1]
Wed, 4 Nov 2026 06:02:38 UTC (569 KB)
来源：arXiv cs.CL · arxiv.org