面向医疗 AI 的多方法评估框架：整合基准测试、LLM 自动评分与专家评审
Towards Multi-Method Evaluation of AI in Health: Integrating Benchmarks, Autoraters, and Expert Evaluation
研究者提出一套面向医疗 GenAI 的结构化多方法评估框架，以 Critical User Journeys（CUJs）锚定评估重点，并映射到基础模型开发、产品开发、部署后监控三阶段生命周期。
Authors:Sarah Montgomery-Taylor, Hamsa Subramaniam, Rob Brisk, Liam Foster, Jack Ching, Andrew Marshall, Jim Taylor, John Hernandez, Jen McKay, Dillon Obika, Nnamdi Ezeanochie, Nira Goren, Tiffany Guo, Rachel Kwon, Arlene Chung, Jacqueline Shreibati, Stella Yeoum, Brett Hatfield, Zelda Gulledge, Ankit Nathu, Ferenc Kazinczi, Yun Liu, Jonathan Krause, Heather Cole-Lewis, Susan Thomas, Megan Jones Bell, Bakul Patel, Joelle Barral, Katherine Chou, Yossi Matias, Dana Gelb Safran, Elizabeth A. McGlynn, Michael Howell
Abstract:General-purpose generative AI (GenAI) models are being rapidly adopted in healthcare, but they are challenging to assess with traditional evaluation frameworks. These conceptual, methodological, and practical challenges parallel decades of work in evaluating clinician and health system quality and safety. We describe the adaptation of methods from several fields, combined with emerging technical approaches, into a structured, multi-method evaluation framework designed to give well-rounded assessments of GenAI in health at scale. The approach first anchors evaluation to Critical User Journeys (CUJs) to help prioritize out of a large choice space, and maps these to a phased AI system lifecycle (base model development, product development, and post-deployment monitoring), since evaluation at each stage requires different methodological approaches. In this context, the approach then integrates three distinct evaluation methods (closed-question benchmarks, LLM-as-a-judge autoraters, and expert human review) across three core domains: accuracy, safety, and usefulness. Taken together, this approach provides a strategic framework for bridging the gap between benchmarking results and performance in the real world with the goal of supporting decision-making and accountability across development, procurement, and deployment for product teams, health systems, and regulators.
| Comments: | 16 pages, 1 figure, 2 tables |
| Subjects: | Computers and Society (cs.CY) |
| Cite as: | arXiv:2611.05127 [cs.CY] |
| (or arXiv:2611.05127v2 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05127 arXiv-issued DOI via DataCite |
Submission history
From: Sarah Montgomery-Taylor [view email]
[v1]
Tue, 3 Nov 2026 16:12:47 UTC (355 KB)
[v2]
Wed, 4 Nov 2026 18:33:13 UTC (355 KB)
来源：arXiv cs.CY · arxiv.org