医疗领域大语言模型评估方法入门指南
A primer on evaluation methods for large language models in healthcare
一篇综述系统梳理了医疗 LLM 评估的四大关键领域：研究设计原则、统计方法、能力评估与临床应用评估。能力评估涵盖多项选择、智能体和多轮 benchmark，以及 token 用量与延迟等运行指标；临床应用评估则涉及自由文本输出准确性验证，如人工审核和 LLM-as-a-judge，以及临床试验方法。文章强调评估方法需与研究问题对齐。
Authors:Suzannah E McKinney, Phuc Vu, Samuel A Justice, Christopher Humphries, Alyssa Pradhan, Timothy J Keyes, Kameron C Black, John L Havlik, David JH Wu, Jack W O'Sullivan, Ethan Goh, Sarah F Mercaldo, James M Hillis
Abstract:Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and not harm. This evaluation can be more challenging than traditional machine learning for many reasons, including probabilistic and open-ended outputs, and behavior that shifts with prompt design and accumulated context. This review covers four key areas of LLM evaluation: principles of study design, statistical methods, capability evaluation and clinical application evaluation. Capability evaluation considers different benchmarks, including multiple-choice, agentic and multi-turn benchmarks, alongside operational metrics like token usage and latency. Clinical application evaluation addresses establishing accuracy of free text outputs, such as human review and LLM-as-a-judge, and clinical trial approaches. Across sections, we describe underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question. Together, this article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.14819 [cs.CL] |
| (or arXiv:2609.14819v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.14819 arXiv-issued DOI via DataCite |
Submission history
From: Suzannah McKinney E [view email]
[v1]
Sun, 13 Sep 2026 22:21:21 UTC (1,976 KB)
[v2]
Thu, 17 Sep 2026 21:38:42 UTC (1,976 KB)
[v3]
Thu, 15 Oct 2026 02:33:07 UTC (11,458 KB)
来源：arXiv cs.CL · arxiv.org