arXiv cs.AI· Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique·· 3 天前AI 评分22
MS-Exam-Gen：面向多发性硬化 MRI 影像知识评估 LLM 的自动基准构建工具包
MS-Exam-Gen: An Automatic Benchmark Construction Toolkit for Evaluating LLMs on Multiple Sclerosis-MRI Imaging Knowledge
AI 导读
MS-Exam-Gen 是一个可复现的自动基准构建工具包，用于评估 LLM 对多发性硬化 MRI 文本知识的掌握，不涉及直接影像判读。它从 66 个来源索引出 4,289 个检索块，生成含 16 个主题、53 个子主题的 3,058 道锁定候选题，在 12 个 LLM 端点上完成 36,696 次预测，准确率跨度达 42.8 个百分点（89.7% 至 46.9%）。
正文 · 原文
Abstract:Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multiple sclerosis MRI (MS-MRI) provides a high-stakes textual-knowledge test case because correct reasoning requires current diagnostic criteria, standardized acquisition and reporting knowledge, longitudinal monitoring concepts, lesion morphology, and recognition of difficult mimics. We present MS-Exam-Gen, a reproducible framework for constructing and auditing a text-based multiple-choice question (MCQ) benchmark for MS-MRI knowledge; it does not evaluate direct MRI image interpretation. MS-Exam-Gen targets source-grounded criteria, protocols, reporting, and differential diagnosis. The framework combines expert-source indexing, exam-oriented topic induction, evidence-grounded MCQ generation, automated quality audits, a same-family consistency screen, and empirical calibration. From a 66-source corpus indexed into 4,289 retrieval chunks, the pipeline produced a locked 3,058-item candidate benchmark spanning 16 topics and 53 subtopics. Evaluation across 12 primary LLM endpoints yielded 36,696 item-level predictions and separated performance over a 42.8-percentage-point accuracy range (89.7% to 46.9%). Across these endpoints, 25.5% of items were missed by at least four. Post-generation audits showed that refreshed construction reduced measurable answer cues, while option-order testing showed that absolute MCQ scores remain position-sensitive. Generated construction labels remain metadata rather than validated psychometric categories. Because expert adjudication and full option-order counterbalancing remain future work, MS-Exam-Gen is not a clinically certified examination. It should be interpreted as an automatically filtered, source-grounded candidate benchmark and reproducible audit workflow for item-level and topic-specific LLM evaluation.
| Comments: | 7 pages, 3 figures. Accepted for publication to BHI 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| MSC classes: | 68T01, 68T50 |
| ACM classes: | I.2.7; H.3.3 |
| Cite as: | arXiv:2610.06170 [cs.AI] |
| (or arXiv:2610.06170v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.06170 arXiv-issued DOI via DataCite |
Submission history
From: Abdul Basit [view email]
[v1]
Mon, 5 Oct 2026 11:41:57 UTC (240 KB)
[v2]
Sat, 10 Oct 2026 12:55:19 UTC (240 KB)
来源：arXiv cs.AI · arxiv.org