arXiv cs.CL· Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo·· 13 小时前AI 评分31
评估大语言模型临床推理能力的评分标准全景：现有工具、缺失维度与整合方向
A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
AI 导读
一篇结构化叙述性综述梳理了医学教育评估工具、2023 年以来的临床 LLM benchmark 与通用长文本生成评估方法，围绕问题表征、时间综合、鉴别与管理推理、反事实推理、校准不确定性、推理忠实性六个维度展开。
正文 · 原文
Abstract:Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integrating and updating evidence across time and sources to form, revise and justify a patient's problem representation and a defensible plan.
This structured narrative review maps three literatures: medical education assessment instruments, clinical LLM benchmarks published from 2023 onwards, and general-domain methods for evaluating long-form generation. We examine six dimensions: problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. Preprints are included and flagged.
No single instrument covers all six dimensions. Problem representation and differential or management reasoning are reasonably covered, although reliability varies by instrument and setting. TIMER-Eval targets temporal synthesis, and ER-Reason assesses sequential diagnostic belief updating. Dedicated uncertainty and counterfactual evaluations are emerging, but their applicability to longitudinal free-text reasoning remains limited. Factual completeness is well theorised in general-domain evaluation, with early clinical evidence of important omissions. Faithfulness remains the weakest dimension, with one identified clinical causal-ablation study on multiple-choice questions.
Existing tools should be combined through binary rubric items, separate completeness and correctness scores, case-specific importance weighting with non-compensable safety caps, temporal order-consistency checks, and chance-corrected reliability reporting. Further design work is needed for calibrated uncertainty, counterfactual reasoning and faithfulness over longitudinal free-text records. This review provides a design rationale, not a validated instrument.
| Comments: | 13 pages, 1 table. Structured narrative review |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.01938 [cs.CL] |
| (or arXiv:2610.01938v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.01938 arXiv-issued DOI via DataCite |
Submission history
From: Zhangshu Joshua Jiang [view email]
[v1]
Thu, 1 Oct 2026 16:07:12 UTC (26 KB)
[v2]
Wed, 21 Oct 2026 14:09:12 UTC (26 KB)
来源：arXiv cs.CL · arxiv.org