跳到正文
arXiv cs.AI· Yipeng Sun, Jiajun Wang, Siyuan Mei, Yan Xia, Siming Bayer, Vincent Christlein, Andreas Maier·· 4 小时前AI 评分34

跨模态对齐的关键因素:LightAlign 用少量图文对实现医学影像对齐

What Matters for Cross-Modal Alignment

AI 导读

研究在自然图像、环境音频、胸部X光片和3D CT四种模态上测试零配对对齐,仅COCO成功,其余四个数据集均失败,原因是模态共变子空间方差占比过小。

正文 · 原文

View PDF HTML (experimental)

Abstract:The Platonic Representation Hypothesis posits that models trained on different modalities converge toward a shared representation. Related work aligns frozen image and text encoders on natural images without paired data. We test zero-pair alignment on five datasets across four modalities: natural images, environmental audio, chest radiographs, and 3D computed tomography (CT). It succeeds only on COCO and fails on the other four. We analyze the failures and find that alignment must first locate the subspace where the modalities co-vary and then orient the mapping within it. Consistent with previous studies, medical imaging is the hardest case: its shared subspace carries the smallest fraction of the variance. When this fraction is small, alignment needs image-text pairs, and random-matrix theory predicts how many pairs are needed. These findings motivate LightAlign: a few pairs let a ridge regression between two frozen encoders locate and orient the mapping simultaneously, and a classifier fitted only on report text then predicts findings from the mapped images. On 3D CT, LightAlign surpasses models trained end to end on image-report pairs without finding-level labels: with only 200 pairs, LightAlign outperforms CT-CLIP on CT-RATE (0.754 vs. 0.733 AUROC), and with only 100 CT-RATE pairs, it does so on RAD-ChestCT, an out-of-distribution dataset from a different hospital. On CheXpert, it reaches 0.862 mean AUROC with only about 5% as many pairs as models trained on MIMIC-CXR.
Comments: Code: this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.20000 [cs.CV]
  (or arXiv:2611.20000v2 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2611.20000

arXiv-issued DOI via DataCite

Submission history

From: Yipeng Sun [view email]
[v1] Sun, 15 Nov 2026 13:18:52 UTC (5,356 KB)
[v2] Tue, 17 Nov 2026 08:04:00 UTC (5,357 KB)

来源:arXiv cs.AI · arxiv.org