arXiv cs.AI· Oscar Freyer, Daniel Glauert, Moritz Hofmann, Max Ostermann, Stephen Gilbert, Jakob Nikolas Kather·· 14 小时前AI 评分31
医学多选题基准测试：13 个决策模型对比 15 种生成式 LLM 配置
Benchmarking decision models against generative large language models on medical multiple-choice questions
AI 导读
研究在 1273 道四选项 MedQA 文本题上对比 13 个决策模型与 15 种生成式 LLM 配置，决策模型准确率 67.71%-89.08%，生成式配置为 78.32%-96.86%。
正文 · 原文
Abstract:Medical workflows often require selecting among predefined options. We compared 13 decision models with 15 generative large language model configurations, including different reasoning settings, using 1,273 four-option MedQA questions under text-only conditions. Accuracy ranged from 67.71% to 89.08% for decision models and from 78.32% to 96.86% for generative configurations. Among the tested configurations, Mercury Decide had the lowest mean final-attempt cost (USD 0.0069 per 1,000 questions), while Clef Omni had the shortest median request duration (0.291 seconds). Decision-model probabilities distinguished correct from incorrect answers, but error rates among answers assigned probabilities of at least 0.90 ranged from 1.71% to 12.77%. Answer agreement across three runs ranged from 92.30% to 100%, while probability repeatability varied. Decision models offered resource advantages alongside lower peak accuracy; selecting models for medical workflows requires considering probability quality and repeatability together with accuracy and the resources needed to detect and correct errors.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38448 [cs.CL] |
| (or arXiv:2610.38448v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38448 arXiv-issued DOI via DataCite |
Submission history
From: Oscar Freyer [view email]
[v1]
Wed, 28 Oct 2026 08:25:35 UTC (4,965 KB)
来源：arXiv cs.AI · arxiv.org