ClinFusion：面向整体医学理解的以视觉为中心的多模态 LLM 系统
ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
ClinFusion 是一个以视觉为中心的多模态 LLM 系统，通过级联空间感知局部融合算子统一 2D 与原生 3D 医学图像理解，并配套 MedIF-Bench 指令遵循评测与基于感兴趣区域的报告生成评测框架。
Authors:Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Sicheng Yang, Lirong Wu, Jinwang Wang, Pengju Wang, Zhitao Zeng, Yizeng Han, Yan Xing, Shengxuan Luo, Tao Feng, Qing Xie, Weigen Yao, Yi Yang, Zuozhu Liu, Jiasheng Tang, Shaocheng Wang, Jitao Wang, Jiahong Dong, Weihua Chen, Feng Xu, Fan Wang
Abstract:Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.
| Comments: | Code: this https URL Models: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.24743 [cs.CV] |
| (or arXiv:2607.24743v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.24743 arXiv-issued DOI via DataCite |
Submission history
From: Hangjie Yuan [view email]
[v1]
Mon, 27 Jul 2026 17:59:49 UTC (2,968 KB)
[v2]
Tue, 28 Jul 2026 10:52:57 UTC (2,960 KB)
[v3]
Sun, 1 Nov 2026 09:43:48 UTC (2,960 KB)
来源：arXiv cs.CL · arxiv.org