arXiv cs.CL· Jeongsoo Choi, Ji-Hoon Kim, Junseok Ahn, Joon Son Chung·· 12 小时前AI 评分38
X-Dubbing：在时间与视觉约束下的语音到语音翻译
X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints
AI 导读
X-Dubbing 将跨语言配音建模为时间感知语音翻译与视频感知语音合成的联合问题，由 Dubbing Translator 生成带时间结构的脚本、Voice Actor 同步渲染，二者共享语言无关的时间预算。该方法无需配对配音数据，两个组件分别基于语音-文本和单语音视频数据训练，并利用合成语音的配音反馈优化翻译器。实验显示其在等时性和音视频一致性等多项指标上优于近期基线。
正文 · 原文
Abstract:Cross-lingual dubbing aims to convert source speech into target-language speech that remains synchronized with the original video. The key to this task is to produce a faithful translation that is naturally speakable within the source timing, which is non-trivial because languages differ in duration, rhythm, and articulation. In this paper, we propose X-Dubbing, which formulates the task as a joint problem of time-aware speech-to-text translation and video-aware text-to-speech. Specifically, our method consists of a Dubbing Translator that generates temporally structured scripts and a Voice Actor that renders them in synchrony with the source video, both grounded in a language-agnostic timing budget. We further refine the translator with dubbing-aware feedback from the synthesized speech, encouraging scripts that yield well-synchronized dubbing. This design requires no paired dubbing data, as the two components are trained on speech-text and monolingual audio-visual data respectively. Extensive experiments show that X-Dubbing outperforms recent baselines across diverse metrics, including isochrony and audio-visual correspondence.
| Subjects: | Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2610.32042 [cs.CL] |
| (or arXiv:2610.32042v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32042 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jeongsoo Choi [view email]
[v1]
Fri, 23 Oct 2026 01:31:30 UTC (181 KB)
来源：arXiv cs.CL · arxiv.org