arXiv cs.AI· Youngtaek Oh, Qiyu Wu, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji·· 9 天前AI 评分23
Syn-Omni：面向全模态嵌入的结构化专精与渐进协作框架
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings
AI 导读
Syn-Omni 是一个面向全模态嵌入的统一框架，通过 Orthogonal Modality-Expert LoRA（OME-LoRA）将适配分解为共享 LoRA 路径与模态专家 LoRA 路径，并用 Progressive Synergy Routing（PSR）让专家先建立模态先验再逐步跨模态协作。
正文 · 原文
Abstract:Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.
| Comments: | Accepted to EMNLP 2026 (Long, Findings). Code: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.12256 [cs.CV] |
| (or arXiv:2610.12256v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12256 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Youngtaek Oh [view email]
[v1]
Thu, 8 Oct 2026 16:26:03 UTC (666 KB)
来源：arXiv cs.AI · arxiv.org