arXiv cs.CL· Rafal Kocielnik, J. Everett Knudsen, Steven Y. Cen, Jasmine Lin, Cherine H. Yang, Atharva Deo, Ujjwal Pasupulety, Peter Wager, Anima Anandkumar, Andrew J. Hung·· 12 小时前AI 评分26
多实例 LLM 框架用于评估外科手术反馈质量
A Multi-Instance LLM Framework for Rating the Quality of Surgical Feedback
AI 导读
研究提出一个两阶段 LLM 框架，通过多实例提示反复独立采样候选标准并整合，结合外科领域知识注入，自动发现 Encouragement、Urgency、Clarity 等可解释的反馈质量评分标准，再用 LLM-as-a-judge 对术中反馈打分。在 4.2k 条带教反馈实例上，AI 发现的标准在预测反馈有效性（包括学员行为调整与带教者认可）上优于以往基于内容的框架。
正文 · 原文
Abstract:Verbal feedback delivered by attending surgeons in the operating room plays a critical formative role in resident trainee skill acquisition. Yet, assessing the quality of trainer feedback and its effectiveness in influencing trainee behavior during live surgery remains a challenge. Prior studies assessed feedback content relying on extensive manual annotation by expert human raters and focused on developing broad taxonomies that overlook the qualitative aspects of feedback delivery such as clarity or urgency. Limited existing automated methods, including keyword analysis and topic modeling, also fail to capture these nuanced aspects. We introduce a two-stage LLM-based framework that discovers interpretable feedback quality criteria grounded in the context of surgical training. Our method uses multi-instance prompting via repeated independent sampling of candidate criteria followed by consolidation, and surgical domain knowledge injection to discover a small set of human interpretable scoring criteria (e.g., Encouragement, Urgency, Clarity). These criteria are then used to automatically score live surgical feedback via an LLM-as-a-judge approach. Evaluation on 4.2k trainer feedback instances demonstrates that our AI-discovered criteria outperform prior content-based frameworks in predicting feedback effectiveness, including observed trainee behavioral adjustments and trainer approval. This work advances scalable, human-aligned assessment of communication quality in the operating room and provides a foundation for improving surgical teaching practices.
| Comments: | 38 pages, 6 figures (1-3 in the body, D1, D2, E1 in the appendices) and 14 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| MSC classes: | 68T50 |
| ACM classes: | I.2.7; J.3; K.3.1 |
| Cite as: | arXiv:2605.25440 [cs.CL] |
| (or arXiv:2605.25440v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.25440 arXiv-issued DOI via DataCite |
Submission history
From: Rafal Kocielnik [view email]
[v1]
Mon, 25 May 2026 05:31:44 UTC (3,415 KB)
[v2]
Sat, 31 Oct 2026 01:14:00 UTC (2,748 KB)
来源：arXiv cs.CL · arxiv.org