arXiv cs.AI· Haenara Shin, Dongjoo Lee, Sungwoo Bae, Kwon Joong Na, Daeseung Lee, Hongyoon Choi·· 12 小时前AI 评分27
VGL-stGen：面向肺部组织学与空间转录组学的共享多模态模型
VGL-stGen: Learning a shared multimodal model for lung histology and spatial transcriptomics
AI 导读
VGL-stGen 是一个从预训练医学视觉语言模型适配而来的多模态大语言模型，使用配对的肺部 H&E 图像与空间转录组数据训练，单个 checkpoint 支持组织与细胞类型分类、基因预测及条件表达补全等十三项任务。任务特定解码可输出分类标签、基因列表和结构化表达水平；固定查询基因身份并自回归预测其水平能改善结构化预测。图像置换实验显示正确配对的 H&E 图像可提供观测表达之外的信息。
正文 · 原文
Abstract:VGL-stGen is a multimodal large language model adapted from a pretrained medical vision-language model using paired lung H&E and spatial transcriptomics with single-cell references. Training integrates image-expression and expression-cell-type pairs, and gene-conditioned pseudo-labels transfer transcriptomic cell-type supervision to histology images. One deployed checkpoint supports thirteen evaluated tasks spanning tissue and cell-type classification, gene prediction and conditional expression completion from combinations of histology, cell identity and observed gene expression. Task-specific decoding produces categorical labels, gene lists and structured expression levels. For conditional completion, fixing queried gene identities and predicting their levels autoregressively improves structured prediction. Controlled adaptation shows that restricting supervision to requested expression-level tokens can further improve image-conditioned completion. Image-permutation experiments indicate that correctly paired histology adds information beyond observed expression. External classification relies partly on computational references with shared annotation lineage, and some H&E sources were exposed during cancer training or model development. Within these conditions, VGL-stGen demonstrates how a medical vision-language backbone can learn from spatial transcriptomics and histology to connect tissue morphology and molecular states through shared generative modeling.
| Comments: | 46 pages, 18 figures, 16 tables; appendix included |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Quantitative Methods (q-bio.QM) |
| Cite as: | arXiv:2610.28266 [cs.CV] |
| (or arXiv:2610.28266v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28266 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongyoon Choi Dr [view email]
[v1]
Tue, 20 Oct 2026 14:22:47 UTC (5,308 KB)
来源：arXiv cs.AI · arxiv.org