Med Evidence Bench（MEB）：大语言模型临床推理的可溯源评测基准
An evidence-traceable benchmark of clinical reasoning in large language models
研究团队构建了 Med Evidence Bench（MEB），包含 1,536 道由医生编写的题目，覆盖指南知识、标准化临床场景和真实实验室场景三个层级，每条评分标准都链接到带版本号的原文片段。
Authors:Zhiqun Zhao (1), Xiaodong Yang (2), Bruce Gu (3 and 4), Ming Li (5), Yi Fu (6), Song Li (7), Qi Mei (8), Zhi Liu (9), Kai Zhang (10), Kexue Fu (3 and 4), Jing Wu (11), Xiaofei Lu (12), Yule Li (1), Zheng Wang (2), Haiquan Lu (1), Xiaoming Wu (3 and 4) ((1) Advanced Medical Research Institute and Key Laboratory for Experimental Teratology of the Ministry of Education, Cheeloo College of Medicine, Shandong University, Jinan, China, (2) Department of Urology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China, (3) Shandong Computer Science Center, National Supercomputer Center in Jinan, Jinan, China, (4) Qilu University of Technology, Shandong Academy of Sciences, Jinan, China, (5) Department of Rheumatology and Immunology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China, (6) Information Network Management Office, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China, (7) Department of Medical Oncology, Qilu Hospital, Shandong University, Jinan, China, (8) Department of Oncology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China, (9) Department of Clinical Laboratory, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China, (10) Department of Breast Surgery, General Surgery, Qilu Hospital, Shandong University, Jinan, China, (11) Department of Central Laboratory, Public Health Clinical Center Affiliated to Shandong University, Shandong University, Jinan, China, (12) Department of Breast Surgery, The First Affiliated Hospital of Shandong First Medical University, Jinan, China)
Abstract:Benchmark scores for medical large language models (LLMs) are difficult to interpret when scoring criteria cannot be traced to clinical evidence. Here we developed Med Evidence Bench (MEB; this https URL), comprising 1,536 physician-constructed items across three levels: guideline knowledge, standardized clinical scenarios and real-world laboratory scenarios. Each criterion stating a medical claim is linked to a versioned source-text span. Across ten LLMs, the Evidence Trace Score (ETS) was 0.975-0.997 for guideline knowledge, 0.292-0.658 for standardized scenarios and 0.182-0.447 for real-world scenarios, and rankings varied across levels. For real-world scenarios, scores for identifying clinical information (mean, 0.770) exceeded those for matching evidence (0.165) or formulating recommendations (0.160). On 1,296 criterion-level judgments, automated grading agreed with the majority vote of three physicians (macro-F1, 0.930; 95% confidence interval, 0.915-0.945), although agreement was lower for evidence application. These findings from Chinese-language materials indicate that guideline recall does not ensure patient-specific evidence use and do not establish clinical effectiveness.
| Comments: | 25 pages, 8 figures, 4 tables. Zhiqun Zhao, Xiaodong Yang and Bruce Gu contributed equally. Corresponding authors: Zheng Wang, Haiquan Lu and Xiaoming Wu |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.13562 [cs.CL] |
| (or arXiv:2611.13562v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13562 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhiqun Zhao [view email]
[v1]
Tue, 10 Nov 2026 04:28:57 UTC (6,415 KB)
来源：arXiv cs.CL · arxiv.org