arXiv cs.AI· Haoxiang Zhao, Bin Wang, Yaofei Wang, Xiao Lan, Yunfeng Diao, Xu Wang·· 14 小时前AI 评分20
UniPRO 研究：多模态视频检索中融合顺序如何影响性能
When Fusion Order Matters: Insights on Progressive Multimodal Representation Learning for Cross-modal Retrieval
AI 导读
研究者提出 UniPRO，用可配置融合顺序、组件与训练设置一致的框架研究多模态视频检索中的融合顺序问题。实验发现逐模态融合优于堆叠融合，且顺序变化会改变性能；在 R@1 上，文本与音频检索偏好把与查询匹配的模态放在最后，训练和测试中人为缺失语音则会反转偏好顺序。据此预测的视觉最后方案在 MSR-VTT-9k 上取得 97.9 R@1，为所评估顺序中最高。
正文 · 原文
Abstract:Multimodal video retrieval combines visual, acoustic, and speech information, yet how their fusion order should depend on the data and the retrieval query remains unclear. We introduce Unified Progressive Representation Learning for Order Analysis, or UniPRO, to investigate this question through configurable fusion schedules with matched model components and training settings. Starting from text-to-video retrieval, we find that the tested one-by-one schedules outperform stacked fusion, while changing the order within one-by-one fusion also changes performance. To guide order selection, we examine two complementary considerations: learning difficulty under unequal modality availability and preservation of information relevant to the query. Complete-modality controls show that missing inputs alone cannot explain order sensitivity. At R@1, text and audio retrieval favor schedules that introduce the modality matching the query last. Controlled speech missingness during training and testing reverses the preferred order in both R@1 and SumR, with different crossover points. These findings motivate balancing the two considerations when selecting a fusion path. We then use them to predict a visual-last schedule for image queries, which achieves 97.9 R@1, the highest among the evaluated orders on MSR-VTT-9k. Main experiments on four text-to-video benchmarks establish the effectiveness of UniPRO, with component ablations assessing its supporting design. Together, the results identify fusion order as a task-dependent choice that should account jointly for modality availability and the retrieval query.
| Subjects: | Artificial Intelligence (cs.AI); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.35125 [cs.AI] |
| (or arXiv:2610.35125v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35125 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haoxiang Zhao [view email]
[v1]
Mon, 26 Oct 2026 10:00:45 UTC (981 KB)
来源：arXiv cs.AI · arxiv.org