arXiv cs.AI· Amama Mahmood, Shiye Cao, Laura Fleig, Junan Jiang, Theodore Jensen, Yee-Yin Choong, Kristen Greene, Chien-Ming Huang·· 10 小时前AI 评分31
医疗场景 AI 记录助手评估碎片化：54 项研究拆出 96 种度量方式
Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare
AI 导读
一项针对医疗场景 ambient AI scribes 的综述发现，54 项近期评估研究共涉及 10 个评估维度和 40 项指标，却被拆成 96 种不同度量方式，其中 35 种（36%）仅被单一研究使用。
正文 · 原文
Abstract:Ambient artificial intelligence (AI) scribes have moved rapidly from experimental pilots to enterprise-wide deployment across health systems, promising reduced documentation burden and improved clinician experience. While research evaluating these tools has expanded quickly, there are no agreed-upon protocols or standardized metrics for assessing their performance, making it difficult to compare findings or cumulatively synthesize evidence across studies. Across 54 recent evaluation studies, we identify ten evaluation dimensions and 40 distinct metrics, operationalized in 96 different ways-35 of them (36%) used by only a single study. Assessment is dominated by documentation efficiency and time, measured in 44 of 54 studies. The other dimensions appear at broadly similar rates (16-24 studies each), and safety and clinical outcome least consistently of all. Ripple effects on the broader care system, such as impact on patients' care outcomes, redistribution of work, changes in billing and coding intensity, and downstream use of notes by other clinicians are rarely assessed. Moreover, the metrics used rely heavily on self-report and are fragmented and inconsistently operationalized, which limits comparability and hinders synthesis across studies. As an effort to support more rigorous and cumulative evaluation, in this paper we review the study designs, data sources, and metrics commonly used to evaluate ambient AI scribes in healthcare settings, discuss the limitations of current evaluation practice, and argue for a more holistic approach grounded in a standardized, multi-dimensional set of metrics.
| Subjects: | Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14284 [cs.HC] |
| (or arXiv:2611.14284v1 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14284 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Amama Mahmood [view email]
[v1]
Tue, 10 Nov 2026 16:26:02 UTC (1,535 KB)
来源：arXiv cs.AI · arxiv.org