临床医生使用语言模型的方式与模型评测方式存在分歧
Clinician use of language models diverges from how the models are evaluated
一项分析 6,342 名临床医护人员在八个月内向机构助手发送的 127,833 条查询发现，文档与行政类占 36.2%、知识检索占 28.9%，诊断仅占 3.7%，超三分之一查询按原样无法得到良好回答。研究者用 RCQ-Map 框架比对 58 个公开基准（Clinical AI Benchmark Atlas），中位基准不含任何文档请求，任务构成与真实使用仅重合 31%。
Authors:Krithik Vishwanath, Haitong Lin, Anton Alyakin, Jin Vivian Lee, D. Brock Hewitt, Jie J. Yao, William Robert Small, Hammad A. Khan, Cordelia Orillac, Aakaash Varma, Brandon Ye, Daniel Alexander Alber, Gustavo Stolovitzky, Batia Wiesenfeld, Oded Nov, Wei Wu, Kang Zhang, Yindalon Aphinyanaphongs, Tim Requarth, Eric Karl Oermann, The International Digital Twin Consortium in Healthcare, Medicine
Abstract:Large language model (LLM) assistants are being deployed to clinicians across health systems, and judgments about their readiness rest largely on benchmark scores, most of them derived from examination questions or curated cases. A benchmark predicts performance in deployment only to the extent that its items resemble real use, yet whether benchmarks reflect the work these systems receive has rarely been measured. Here we analyze 127,833 queries sent by 6,342 physicians, advanced practice providers and nurses in 35 specialties to an institutional assistant during an eight-month roll-out. We characterize each query with RCQ-Map, a clinician-validated framework grounded in taxonomies of clinical questions and of LLM evaluation, which records its task, intent, answerability, missing information and potential harm. Documentation and administration (36.2%) and knowledge retrieval (28.9%) made up nearly two-thirds of use, and diagnosis 3.7%; more than a third of queries could not be answered well as posed. Applying RCQ-Map to 58 public benchmarks drawn from major evaluation suites and frontier model reports, which we assemble into the Clinical AI Benchmark Atlas, showed that the median benchmark contained no documentation requests and shared 31% of the task mix of real use, less than an even spread across task categories would. Benchmarks in suites designed to resemble clinical practice were individually no closer to real use than those used in frontier model reports. Benchmark scores therefore say little about how clinical AI performs on most of the work it is actually given, and evaluation should be matched to real clinical use.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.11069 [cs.CL] |
| (or arXiv:2610.11069v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11069 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Krithik Vishwanath Mr. [view email]
[v1]
Thu, 8 Oct 2026 01:29:04 UTC (890 KB)
来源：arXiv cs.CL · arxiv.org