arXiv cs.CL· Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra, Robert E. Tillman·· 13 小时前AI 评分31
MedicalBench：评估大语言模型改进医疗概念抽取的新基准
MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction
AI 导读
MedicalBench 是首个针对隐式、带证据锚定医疗概念抽取的系统性基准，基于 MIMIC-IV 出院小结与人工核验的 ICD-10 编码构建，包含隐式正例、语义易混负例及 LLM 判断与专家评估不一致的案例。
正文 · 原文
Abstract:Medical concept extraction from electronic health records underpins many downstream applications, yet remains challenging because medically meaningful concepts are frequently implied rather than explicitly stated in medical narratives. Existing benchmarks with human-annotated evidence spans underscore the importance of grounding extracted concepts in medical text. However, they predominantly focus on explicitly stated concepts instead of implicit concepts. We present MedicalBench, a benchmark for medical concept extraction with evidence grounding that evaluates implicit medical reasoning. MedicalBench formulates medical concept extraction as a verification task over medical note-concept pairs, coupled with sentence-level evidence identification. Built from MIMIC-IV discharge summaries and human-verified ICD-10 codes, the dataset is curated through a multi-stage large language model (LLM) triage pipeline followed by medical annotation and expert review. It deliberately includes implicit positives, semantically confusable negatives, and cases where LLM judgments disagree with medical expert assessments. We define two complementary evaluation tasks: (1) medical concept extraction and (2) sentence-level evidence retrieval, enabling assessment of both correctness and interpretability. Benchmarking state-of-the-art LLMs reveals that performance remains modest, highlighting the difficulty of extracting implicitly expressed concepts. We further show that performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders. MedicalBench provides the first systematic benchmark for implicit, evidence-grounded medical concept extraction, offering a foundation for developing medical language models that can both identify medically relevant concepts and justify their predictions in a transparent and medically faithful manner.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.20197 [cs.CL] |
| (or arXiv:2605.20197v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.20197 arXiv-issued DOI via DataCite |
Submission history
From: Sanjit Singh Batra [view email]
[v1]
Sun, 5 Apr 2026 14:11:00 UTC (716 KB)
[v2]
Wed, 21 Oct 2026 01:06:06 UTC (923 KB)
来源：arXiv cs.CL · arxiv.org