arXiv cs.CY· Daniel S. Epstein·· 12 小时前AI 评分42
AusClinBench：面向澳大利亚现行临床政策的大语言模型确定性评分基准
AusClinBench: a deterministically scored benchmark of large language models against current Australian clinical policy
AI 导读
研究者构建了 AusClinBench 基准，从澳大利亚指南与目录中编写 142 道题，覆盖 PBS 授权、MBS 计费、Ahpra 义务、分诊红旗、免疫与筛查五个领域，每题仅一个可验证答案并按精确匹配评分。
正文 · 原文
Abstract:AI models already inform clinical decisions and draft notes in Australian clinical software, usually without the practitioner being told which model is used. Existing medical benchmarks are written for other jurisdictions. We wrote 142 questions from Australian guidelines and schedules across five domains (PBS authority, MBS billing, Ahpra obligations, triage red flags, immunisation and screening), each with one verifiable answer scored by exact match. Each question ran blind and with the governing clause supplied, which separates failure to recall the current rule from failure to apply it. We tested 28 models between July and October 2026. Blind accuracy ranged from 51.5% to 92.7%, and the highest-scoring models could not be separated. Of 791 failed model-question pairs, 684 (86%) were wrong blind and right with the clause. Confident-error rates ranged from 6.6% to 47.8%. On five immunisation questions where the guideline gives no directive, 82% of answers stated one. On 20 Aboriginal and Torres Strait Islander health questions, models did no worse overall, but on a few the wrong answers gave the rule for the general population, an older rule or another country's rule. We propose treating the model as an ingredient of a clinical product, with a minimum standard on local rules that sits beside product evaluation. The keys are one general practitioner's and are not yet independently reviewed. The benchmark measures base models, not clinical products, and does not license any model for clinical use.
| Comments: | 19 pages, 3 figures. Technical report, version 1.4.1 of the benchmark. Not peer reviewed. Answer keys not yet independently reviewed. Results and methods: this https URL |
| Subjects: | Computers and Society (cs.CY); Computation and Language (cs.CL) |
| ACM classes: | I.2.7; J.3 |
| Cite as: | arXiv:2610.31993 [cs.CY] |
| (or arXiv:2610.31993v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31993 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Daniel Epstein [view email]
[v1]
Thu, 22 Oct 2026 23:25:04 UTC (510 KB)
来源：arXiv cs.CY · arxiv.org