DoseBench：评估 LLM 在非处方药剂量问答中的决策能力

View PDF HTML (experimental)

Abstract:Large language models (LLMs) are increasingly used to answer everyday health questions online, including whether someone can take another dose of an over-the-counter (OTC) medication. Yet this common safety-relevant setting remains underexplored in medical QA. Answering these questions requires checking previous doses, timing, and product-specific dosing limits, while recognizing when important information is missing. We introduce DoseBench, a focused benchmark of 230 consumer-style acetaminophen and ibuprofen dosing questions with human-annotated YES, NO, and AMBIGUOUS labels, and evaluate eight LLMs with five responses per question. Overall accuracy ranges from 44.8% to 62.2%, while models incorrectly approve 14.0% to 60.5% of prohibited doses. Models with similar accuracy can therefore make very different dosing decisions. Additional analyses show that these failures are not easily fixed by giving models more guidance. Asking models to check dose, timing, and missing information makes them more cautious, while providing drug facts improves accuracy but can make them overlook missing information. These findings highlight a broader challenge in health QA: applying rules correctly and recognizing when information is missing.

Comments:	12 pages, 5 figures
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2606.04262 [cs.CL]
	(or arXiv:2606.04262v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2606.04262 arXiv-issued DOI via DataCite

Submission history

From: Maroof Kousar [view email]
[v1] Tue, 2 Jun 2026 22:30:15 UTC (2,556 KB)
[v2] Thu, 29 Oct 2026 00:02:17 UTC (132 KB)