arXiv cs.CY· Miguel Fernandez-de-Retana, Unai Zulaika, Ruben Sanchez-Corcuera, Aritz Bilbao-Jayo, Aitor Almeida·· 13 小时前AI 评分40
开源语言模型能通过西班牙MIR住院医师考试吗？8-27B模型的不确定性、推理与图像使用研究
Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination
AI 导读
研究用西班牙MIR住院医师考试规则评测4个8-27B开源模型共1552道题，三个较大模型若全部作答可排进约前600至3000名，仍有41-43个专科可选。弃答从不划算，因为即使最不自信的答案正确率也高于25%的盈亏平衡点。医学微调以准确率下降为代价降低过度自信，却未改善置信度区分对错的能力；13-38%的正确答案以临床不成立的理据排除干扰项，移除或替换所给图像后准确率不变。
正文 · 原文
Abstract:Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the images it is given. We examine these questions for four open models of 8 to 27 billion parameters on the Spanish MIR residency examination, scoring 1,552 questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first 600 to 3,000 candidates, with 41 to 43 specialties still open to them. Abstention, however, never pays, because even their least confident answers are right more often than the 25% break-even. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, 13-38% of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect, and removing or swapping the provided contextual images left accuracy unchanged.
| Comments: | 13 pages, 8 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG) |
| MSC classes: | 68T50 (Primary) 68T07, 68T37, 92C50 (Secondary) |
| ACM classes: | I.2.7; I.2.1; J.3 |
| Cite as: | arXiv:2610.31586 [cs.CL] |
| (or arXiv:2610.31586v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31586 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Miguel Fernandez-De-Retana [view email]
[v1]
Thu, 22 Oct 2026 17:23:11 UTC (413 KB)
来源：arXiv cs.CY · arxiv.org