炎症性肠病全切片图像与临床数据的多模态融合方法对比研究
Multimodal Fusion Methods for Whole-Slide Images and Clinical Data in Inflammatory Bowel Disease
研究提出模块化框架，将冻结基础模型patch嵌入经注意力聚合得到的切片级嵌入与每次就诊的临床嵌入，输入15种可互换融合方法，在炎症性肠病前瞻性队列中预测STRIDE-II对齐的缓解结局。最佳融合方法macro-F1达0.57-0.75，在六种场景中的四种优于单模态模型，且表现随结局和输入而异。图像依赖程度随缓解类型变化，组织学缓解最高、内镜缓解最低，准确率相近的方法对切片的依赖程度差异很大。
Authors:Ankana Banerjee (1), Mark van der Giezen (1, 2, 3), Ole Gunnar Aasprong (2), Emiel A. M. Janssen (1, 2), Øyvind Steinsbø (2, 4), Tore Grimstad (2, 4), Kjersti Engan (1) ((1) University of Stavanger, Norway, (2) Stavanger University Hospital, Norway, (3) University of Greenwich, UK, (4) University of Bergen, Norway)
Abstract:Multimodal AI fusion of histology and clinical data is an active research area in computational pathology, but fusion methods are rarely compared under identical conditions and across different clinical endpoints. We present a modular framework in which slide-level embeddings, obtained by training an attention-based aggregator over patch embeddings from a frozen foundation model, and per-visit clinical embeddings are fed into 15 interchangeable fusion methods with a shared output dimension and prediction head (except decision-level methods). Using a prospective cohort of newly diagnosed inflammatory bowel disease patients, we predicted STRIDE-II-aligned remission outcomes from baseline or follow-up visits. The best fusion method reached macro-F1 of 0.57-0.75, outperforming both single-modality models in four of six scenarios, and differed by outcome and input. Image reliance, the drop in balanced accuracy when the slide embedding is removed, tended to follow the remission type - highest for histological and lowest for endoscopic remission; methods with similar accuracy relied on the slide to very different degrees.
| Comments: | 5 pages, 4 figures, Submitted to the 2027 IEEE 24th International Symposium on Biomedical Imaging (ISBI) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.05579 [cs.CV] |
| (or arXiv:2611.05579v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05579 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ankana Banerjee [view email]
[v1]
Tue, 3 Nov 2026 20:11:07 UTC (4,655 KB)
来源：arXiv cs.AI · arxiv.org