arXiv cs.AI· Andrea Agostini, Simon B\"ohi, Moritz Vandenhirtz, Samuel Ruiperez-Campillo, Max Kr\"ahenmann, Silke M\"uhlstedt, Irene Cannistraci, Ece \"Ozkan Elsen, Julia E. Vogt, Thomas M. Sutter·· 9 天前AI 评分24
LAMAE：面向多模态心脏表征学习的潜在注意力掩码自编码器
Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning
AI 导读
LAMAE 是一种多模态、结构感知的掩码自编码器，通过共享潜在注意力模块在 study-view-entity 层级上直接交换信息，实现患者级表征的联合自监督预训练。该模型在超 50 万次 MIMIC-IV 住院数据上预训练，在院内死亡率、ICD-10 与 DRG 编码、住院时长等多模态任务上优于模态特定预训练及对比学习、视觉语言基线，同时在单模态任务上保持竞争力。
正文 · 原文
Abstract:Cardiovascular diagnosis and treatment rest on integrating complementary modalities, such as electrocardiogram, echocardiography, and chest X-rays, each capturing distinct but complementary aspects of cardiac pathophysiology. Yet most medical foundation models remain modality-specific, combining modalities only for finetuning or post-training. This discards the cross-modal evidence clinicians naturally integrate and ignores the structure within each modality. We introduce Latent-Attention Masked Autoencoder (LAMAE), a multimodal, structure-aware masked autoencoder that jointly learns patient-level representations during self-supervised pretraining. Instead of fusing modalities post hoc, LAMAE exchanges information directly in the latent space through a shared latent-attention module operating over a study-view-entity hierarchy, enabling aggregation of variable observations and handling of missing modalities. Pretrained on over 500'000 MIMIC-IV hospital stays, LAMAE outperforms modality-specific pretraining and strong contrastive and vision-language baselines across multimodal hospital-stay tasks, such as in-hospital mortality, ICD-10 and DRG coding, and length of stay, while remaining competitive on unimodal tasks. Modeling this structure also pays off within a single modality: even without cross-modal information, the latent-attention module improves representations over modality-specific pretraining.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.12035 [cs.AI] |
| (or arXiv:2609.12035v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.12035 arXiv-issued DOI via DataCite |
Submission history
From: Andrea Agostini [view email]
[v1]
Thu, 10 Sep 2026 17:49:44 UTC (1,346 KB)
[v2]
Fri, 2 Oct 2026 15:33:30 UTC (2,702 KB)
[v3]
Thu, 8 Oct 2026 14:39:26 UTC (2,701 KB)
来源：arXiv cs.AI · arxiv.org