arXiv cs.CL· Federica Bologna, Jean-Philippe Corbeil, Matthew Wilkens, Asma Ben Abacha·· 2 天前AI 评分22
面向跨模态与跨解剖部位的放射学报告评估：更对齐的 LLM 评判模型研究
Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies
AI 导读
研究系统评估了 LLM 作为放射学报告评判模型在多种成像模态和解剖部位上的表现，提出新指标 VERT，其与放射科医生判断的 Kendall 相关性较 GREEN 最高提升 26%。在现有指标中 GREEN 最稳健；对 Qwen3-30B 进行集成和轻量微调分别带来最高 9.9% 和 25% 的提升，后者延迟降低 37.2 倍。误差分析显示这些指标虽整体与专家评分相关，但易在遗漏类错误上计数失准。
正文 · 原文
Abstract:Current work on radiology report evaluation has focused primarily on LLM-based metrics for chest X-rays (CXR), leaving open whether such approaches remain robust across other modalities and anatomies, and which model and prompt configurations are best suited to serve as LLM judges. We present a comprehensive evaluation of LLM judges for radiology report evaluation across diverse model families, prompting strategies, and adaptation methods, leveraging two expert-annotated datasets that span multiple imaging modalities and anatomical regions. Specifically, we compare four existing metrics (RadFact, GREEN, FineRadScore, and CRIMSON) with our proposed metric, VERT, across open- and closed-source models of varying sizes and reasoning capabilities. Among existing metrics, GREEN proves the most robust, outperforming more recent CXR-specific approaches. VERT further improves Kendall's correlation with radiologist judgment by up to 26% over GREEN, while ensembling and lightweight fine-tuning of Qwen3-30B yield gains of up to 9.9% and 25%, respectively, of which the latter also has a latency reduction of 37.2x. A systematic error analysis further shows that, while these metrics correlate with expert ratings overall, they tend to miscount errors such as omissions. Overall, our results show that carefully selected general-purpose LLM judges provide strong radiology report evaluation across modalities, with lightweight adaptation offering complementary gains when annotated data is available.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.03376 [cs.AI] |
| (or arXiv:2604.03376v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.03376 arXiv-issued DOI via DataCite |
Submission history
From: Federica Bologna [view email]
[v1]
Fri, 3 Apr 2026 18:10:21 UTC (1,287 KB)
[v2]
Wed, 14 Oct 2026 18:45:08 UTC (1,583 KB)
来源：arXiv cs.CL · arxiv.org