EHRNote-ChatQA：面向纵向出院小结的循证多轮临床问答基准
EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries
EHRNote-ChatQA 是首个基于患者多份出院小结的循证多轮临床问答基准，含 967 个患者级多轮样本、16,072 条经 12 位医学专家逐条审核的 QA 对，覆盖 8 个临床类别。对 24 个闭源与开源权重 LLM 的评测显示，模型在证据定位上比内容作答更吃力，多轮错误会逐轮累积，单轮临床问答表现无法可靠迁移到该场景。
Authors:Jiyoun Kim, Muhan Yeo, Eunhye Jang, Jeewon Yang, Hangyul Yoon, Su Ji Lee, Hee Jo Han, Hee-Jae Jung, Doyun Kwon, Jun young Lee, Jaehun Lee, Jung-Oh Lee, Sunjun Kweon, Jong Hak Moon, Daseul Kim, Minjae Cho, Edward Choi
Abstract:Discharge summaries are crucial clinical documents containing the context of a patient's overall hospital stay, and are routinely reviewed by medical experts for patient readmission, ongoing care, and diagnostic decision-making. When reviewing them, medical experts often must iteratively synthesize information across multiple discharge summaries while verifying the evidence supporting each answer. Although large language models (LLMs) are increasingly explored for clinical question answering, existing benchmarks do not sufficiently reflect this setting: they often evaluate exam-style medical knowledge or focus on single-turn question answering with limited evidence-grounding evaluation. We introduce EHRNote-ChatQA, the first benchmark for evidence-grounded multi-turn clinical question answering over patients' multiple discharge summaries. Built from de-identified MIMIC-IV discharge summaries, EHRNote-ChatQA contains 967 patient-level multi-turn samples spanning one to five notes and 16,072 medical-expert-verified QA pairs (8,036 content questions, each paired with an evidence-grounding question) across eight clinical categories. The benchmark is constructed through an expert-informed pipeline combining discharge-summary structuring schema, expert-curated multi-turn QA templates, and LLM-based generation, followed by review and revision of every single QA sample by 12 medical experts (11 physicians and 1 nurse). Benchmarking 24 proprietary and open-weight LLMs reveals several challenges, including that LLMs struggle more with evidence grounding than content answering, multi-turn errors compound across turns, and single-turn clinical QA performance does not reliably transfer to this setting. These findings establish EHRNote-ChatQA as a rigorous and practical benchmark for evaluating clinical QA systems.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.15735 [cs.CL] |
| (or arXiv:2606.15735v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.15735 arXiv-issued DOI via DataCite |
Submission history
From: Jiyoun Kim [view email]
[v1]
Sun, 14 Jun 2026 10:35:19 UTC (3,211 KB)
[v2]
Tue, 16 Jun 2026 10:33:24 UTC (3,208 KB)
[v3]
Wed, 21 Oct 2026 15:46:36 UTC (3,276 KB)
来源：arXiv cs.CL · arxiv.org