跳到正文
arXiv cs.AI· Sneha Maurya, Spandana Govindgari, Girish Kumar·· 4 小时前AI 评分38

WHBench:用专家在环验证评估前沿 LLM 的女性健康话题表现

WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics

AI 导读

研究者发布女性健康基准 WHBench,包含 10 个女性健康主题下 47 个专家设计的场景,用 23 项评分标准评估 22 个模型,覆盖临床准确性、安全性、公平性等维度。在 3,102 次作答中无模型平均分超过 75%,最佳模型仅达 72.1%,且各模型完全正确率偏低、危害率差异明显。该基准已公开,用于追踪女性健康 AI 更安全、更公平的进展。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large language models are increasingly used for medical guidance, but women's health remains under-evaluated in benchmark design. We present the Women's Health Benchmark (WHBench), a targeted evaluation suite of 47 expert-crafted scenarios across 10 women's health topics, designed to expose clinically meaningful failure modes including outdated guidelines, unsafe omissions, dosing errors, and equity-related blind spots. We evaluate 22 models using a 23-criterion rubric spanning clinical accuracy, completeness, safety, communication quality, instruction following, equity, uncertainty handling, and guideline adherence, with safety-weighted penalties and server-side score recalculation. Across 3,102 attempted responses (3,100 scored), no model mean performance exceeds 75 percent; the best model reaches 72.1 percent. Even top models show low fully correct rates and substantial variation in harm rates. Inter-rater reliability is moderate at the response label level but high for model ranking, supporting WHBench utility for comparative system evaluation while highlighting the need for expert oversight in clinical deployment. WHBench provides a public, failure-mode-aware benchmark to track safer and more equitable progress in womens health AI.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
Cite as: arXiv:2604.00024 [cs.CL]
  (or arXiv:2604.00024v3 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2604.00024

arXiv-issued DOI via DataCite

Submission history

From: Spandana Govindgari [view email]
[v1] Wed, 11 Mar 2026 06:21:37 UTC (1,258 KB)
[v2] Thu, 23 Jul 2026 23:19:26 UTC (1,258 KB)
[v3] Tue, 17 Nov 2026 04:47:58 UTC (1,258 KB)

来源:arXiv cs.AI · arxiv.org