arXiv cs.CL· Xin Cheng, Yuyue Wang, Xihua Wang, Yihan Wu, Kaisi Guan, Yijing Chen, Peng Zhang, Xiaojiang Liu, Meng Cao, Ruihua Song·· 12 小时前AI 评分34
ViSSFlow：用统一流匹配框架联合生成视频条件音效与语音
ViSSFlow: Unifying Video-Conditioned Sound and Speech Generation via Joint Learning
AI 导读
ViSSFlow 是一个统一流匹配框架，将视频到音效（V2S）与视觉文本转语音（VisualTTS）两类任务合并到单一模型中。它通过解耦条件聚合机制，在 Diffusion Transformer 中用 cross-attention 建模语义条件、self-attention 捕捉时间密集的视觉线索，并证明联合优化不会造成任务干扰。
正文 · 原文
Abstract:Video-conditioned audio generation, including Video-to-Sound (V2S) and Visual Text-to-Speech (VisualTTS), has traditionally been studied as two separate tasks despite their shared objective of generating audio conditioned on visual content. Existing methods rely on task-specific architectures and training paradigms, while the possibility of a unified generative framework remains largely unexplored due to heterogeneous conditioning signals and limited joint training data. In this paper, we propose ViSSFlow, a unified flow-matching framework for both V2S and VisualTTS. To effectively integrate diverse conditioning modalities within a Diffusion Transformer (DiT), we introduce a disentangled condition aggregation mechanism that exploits the complementary roles of attention layers, using cross-attention to model semantic conditions and self-attention to capture temporally dense visual cues. Furthermore, contrary to the common assumption that joint optimization of V2S and VisualTTS causes task interference, we show that a single model can be trained end-to-end without sacrificing task-specific performance. To address the scarcity of paired training data covering both sound and speech generation, we further propose a simple yet effective feature-level data synthesis strategy that enables robust joint learning. Extensive experiments on V2S, VisualTTS, and joint generation benchmarks demonstrate that ViSSFlow consistently outperforms state-of-the-art task-specific baselines while providing a unified solution for video-conditioned audio generation. Code and demos are provided in Supplementary Material.
| Comments: | Paper Under Review |
| Subjects: | Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD) |
| Cite as: | arXiv:2509.24773 [eess.AS] |
| (or arXiv:2509.24773v5 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2509.24773 arXiv-issued DOI via DataCite |
Submission history
From: Xin Cheng [view email]
[v1]
Mon, 29 Sep 2025 13:38:24 UTC (2,910 KB)
[v2]
Tue, 30 Sep 2025 05:16:17 UTC (2,910 KB)
[v3]
Tue, 10 Mar 2026 03:06:55 UTC (2,315 KB)
[v4]
Fri, 20 Mar 2026 03:36:49 UTC (2,315 KB)
[v5]
Mon, 2 Nov 2026 02:53:41 UTC (1,283 KB)
来源：arXiv cs.CL · arxiv.org