arXiv cs.CL· Zhisheng Zheng, David Harwath·· 11 小时前AI 评分34
TransDubber：学习保留时间线的语音到语音翻译
Learning Timeline-Preserving Speech-to-Speech Translation
AI 导读
TransDubber 是一个端到端自回归语音到语音翻译系统，可直接从对齐的目标波形中学习时间结构，无需推理时的帧级时间控制或事后时间校正。研究构建了 642K 条严格时长匹配的英中语音对，结合渐进式监督课程与验证器引导的强化学习。
正文 · 原文
Abstract:Speech-to-speech translation (S2ST) increasingly preserves what is said and how it sounds, but not necessarily when speech and silence occur. For dubbing and other temporally grounded applications, matching total duration is not enough: speech and pauses must also occur at corresponding moments. We formulate timeline-preserving S2ST, which aims to preserve the source endpoint and salient internal pauses on a shared absolute timeline. We present TransDubber, an end-to-end autoregressive system that learns this temporal structure directly from aligned target waveforms, without inference-time frame-level timing control or post-hoc temporal correction. To provide this supervision, we construct 642K exactly duration-matched English-Chinese speech pairs by translating and synthesizing target speech on source-defined temporal scaffolds. Training combines a progressive supervised curriculum with verifier-guided reinforcement learning to balance translation quality and temporal alignment. On a 4,353-utterance test set spanning read and spontaneous speech, TransDubber increases the proportion of translations within 2% of the source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550 compared with UniSS, while maintaining competitive translation quality. Our analysis shows that aligned supervision establishes timeline following, while reinforcement learning refines temporal precision and partially offsets the translation-quality cost of strict synchronization. Together, these results demonstrate the effectiveness of our data and training framework for timeline-preserving S2ST. Audio samples, code, datasets, and pretrained models are available at this https URL.
| Comments: | Working in progress |
| Subjects: | Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Sound (cs.SD) |
| Cite as: | arXiv:2611.00690 [eess.AS] |
| (or arXiv:2611.00690v1 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00690 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhisheng Zheng [view email]
[v1]
Sat, 31 Oct 2026 06:39:04 UTC (1,018 KB)
来源：arXiv cs.CL · arxiv.org