arXiv cs.AI· Jinrui Wu, Yun Peng·· 10 小时前AI 评分29
ProtoBridge-CLIP:用观察掩码发现原型做医学图像—报告表征学习
ProtoBridge-CLIP: Observation-Masked Finding Prototypes for Medical Image--Report Representation Learning
AI 导读
ProtoBridge-CLIP 将对称的全局图像—报告对齐与观察掩码的发现监督、报告已确认概念的软关系目标结合,把文本衍生的发现原型与局部视觉 token 对齐,并将不确定和未提及概念排除在直接发现标签之外。
正文 · 原文
Abstract:Radiology reports distinguish affirmed findings, explicit negations, and unmentioned concepts, but conventional global image--report contrastive learning often compresses these different evidence states into a single study-level target. We introduce \model{}, which combines symmetric global image--report alignment with observation-masked finding supervision and a soft relation objective over affirmed report concepts. Text-derived finding prototypes are aligned with local visual tokens, while uncertain and unmentioned concepts are excluded from direct finding labels. We evaluate the framework on three CT benchmarks and five chest-radiograph benchmarks, together with component ablations, paired and clinical retrieval, localization, perturbation robustness, calibration, and per-finding analyses. \model{} achieves macro-AUROC values of 73.6%, 66.5\%, and 68.8% on CT-RATE, RAD-ChestCT, and INSPECT, and 80.2%, 91.4%, 78.5%, 86.5%, and 90.8% on ChestXray14, CheXpert, PadChest, RSNA, and SIIM-ACR, respectively. Relative to the matched global-only controls, the findings and relation objectives yield consistent gains across the evaluated datasets. These results demonstrate the practical value of treating observed and unobserved report concepts differently during medical image--text representation learning.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24002 [cs.CV] |
| (or arXiv:2611.24002v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24002 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Peng Yun [view email]
[v1]
Wed, 18 Nov 2026 07:52:57 UTC (139 KB)
来源:arXiv cs.AI · arxiv.org