arXiv cs.CL· Prithvi Dixit, Pedram Hosseini·· 10 小时前AI 评分34
医学基准评分是否可靠?RIFT 框架揭示临床评测中的评分标准失效模式
Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks
AI 导读
研究将全局评分标准失效分类框架 RIFT 应用于 HealthBench Professional、LiveMedBench 和 MentalHealthBench 三个临床基准,LLM 评审在 HealthBench Professional 上标记 29.6% 的标准为非原子性、65.4% 为错位或僵化。
正文 · 原文
Abstract:Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative. We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected. We apply RIFT, a global rubric failure taxonomy, to three clinical benchmarks (HealthBench Professional, LiveMedBench, and MentalHealthBench), and find failure modes are meaningful: on HealthBench Professional an LLM judge flags 29.6% of criteria as non-atomic and 65.4% as misaligned/rigid, although these rates vary severalfold across five detection judges. Then, we show that these flaws are meaningful and not simply cosmetic. As an example, rewriting bundled criteria of the form "at least one of / all of the following" as equally weighted children and regrading identical responses shifts scores by up to 15.9 percentage points on affected conversations, with disjunctive bundles inflating scores and conjunctive bundles deflating them. Across six answering models and two common graders, these shifts leave the model ranking unchanged, and their net direction depends on how many bundled criteria are penalties. We also find that RIFT can under-detect bundling on clinical rubrics, by an amount that depends on the detection judge: on the same LiveMedBench criteria, judges flag 3.8-15.9% as non-atomic against an estimated 7-19% that are genuinely bundled.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.16023 [cs.CL] |
| (or arXiv:2609.16023v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.16023 arXiv-issued DOI via DataCite |
Submission history
From: Prithvi Dixit [view email]
[v1]
Thu, 10 Sep 2026 01:18:39 UTC (701 KB)
[v2]
Wed, 18 Nov 2026 20:33:26 UTC (1,113 KB)
来源:arXiv cs.CL · arxiv.org