arXiv cs.AI· Shubham Gupta, Siva Reddy, Perouz Taslakian, Valentina Zantedeschi, Cem Subakan·· 12 小时前AI 评分31
用单纯形约束表示缩小 CLIP 与 CLAP 的多模态鸿沟
Closing the Modality Gap via Simplex-Constrained Representations
AI 导读
研究提出将多模态对比模型的嵌入域从单位球面 S^{d-1} 改为概率单纯形 Δ^{d-1}，借助非负、单位质量约束作为跨模态校准，把两种模态锚定到同一坐标系。方法通过轻量 softmax 适配器与 Total Variation 相似度，在五个多模态检索基准上将质心 ℓ2 差距缩小 97–99%，检索性能匹配或超过标准无约束模型。
正文 · 原文
Abstract:While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent \emph{modality gap}. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, $S^{d-1}$, to the probability simplex, $\Delta^{d-1}$. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid $\ell_2$ gap by 97--99\% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.32890 [cs.AI] |
| (or arXiv:2610.32890v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32890 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shubham Gupta [view email]
[v1]
Fri, 23 Oct 2026 16:57:48 UTC (90 KB)
来源：arXiv cs.AI · arxiv.org