arXiv cs.AI· Rotem Gershon, Seffi Cohen, Dean Ariel, Havatzelet Bilavsky-Yarden, Amiel Sberro, Shai Yitzhaki, Osnat Tausky, Naama Golan, Maayan Rabinovitch, Ariel Avraham Hasidim, Itay Arianu, Maya Dagan, Ran Balicer, Noa Dagan·· 13 小时前AI 评分38
决策语言模型 Jev 可作为临床生成式 AI 的低成本快速评判替代方案
Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI
AI 导读
决策语言模型 Jev 在四项临床 AI 评测任务中与前沿及开源 LLM 评判模型对比，在 118 份儿科回答上与医生序数质量评分的一致性（Gwet's AC2 0.50）达到或超过 12 个 LLM 评判模型中的 7 个，包括 Claude Opus 5.5（0.49）和 GPT-6（0.38）。
正文 · 原文
Authors:Rotem Gershon, Seffi Cohen, Dean Ariel, Havatzelet Bilavsky-Yarden, Amiel Sberro, Shai Yitzhaki, Osnat Tausky, Naama Golan, Maayan Rabinovitch, Ariel Avraham Hasidim, Itay Arianu, Maya Dagan, Ran Balicer, Noa Dagan
Abstract:Automated judges of clinical artificial intelligence (AI) outputs are necessary, but large language model (LLM) judges are costly and slow at scale. We compared Jev, a decision language model that returns structured judgments, with frontier and open-weight LLM judges in four use cases. On 118 pediatric answers, Jev's agreement with physicians' ordinal quality ratings (Gwet's AC2 0.50; expected score 0.53) matched or exceeded that of seven of 12 LLM judges, including frontier models, Claude Opus 5.5 (0.49) and GPT-6 (0.38). On physician-graded binary HealthBench criteria, Jev matched or exceeded four of six LLM judges. On MedHELM's open-ended tasks, Jev's agreement with a three-LLM jury approached or exceeded the jurors' internal agreement. On HealthBench Professional, Jev matched the benchmark's reference judge, GPT-5.4, on 91% of criteria (other judges, 91-93%). Frontier judges cost 73-294 times more than Jev and took 7-57 times longer. Jev may enable efficient clinical AI evaluation at scale.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.28826 [cs.AI] |
| (or arXiv:2610.28826v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28826 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Seffi Cohen [view email]
[v1]
Tue, 20 Oct 2026 18:46:47 UTC (2,804 KB)
来源：arXiv cs.AI · arxiv.org