arXiv cs.CL· Ryosuke Suzuki·· 2 天前AI 评分31
用大语言模型让医疗器械文档评估的每个阶段都实现确定性
Making every stage of medical device document assessment with large language models deterministic
AI 导读
研究测试了 14 个商用大语言模型部署（来自 3 家提供商的 7 个模型、6 种调用路径），在各自接受的确定性采样设置下重复同一请求，每个部署都至少有一次返回不同评估结果，其中 12 个出现标签变化。
正文 · 原文
Abstract:To show that a medical device complies with regulation, experts judge whether documents and records meet each requirement and record the judgment for others to verify. For each assessment item that experts define, large language models could prepare a preliminary assessment: a label such as Satisfied or Not satisfied, a reason and quoted evidence. We sent fourteen commercial deployments of such models (seven models from three providers, through six routes) the same request repeatedly, with the deterministic sampling settings each accepted; every deployment returned differing assessments for at least one request, and in twelve the label changed. Reducing reasoning where the interface allows it did not reduce the variation, and prompts of retrieved passages removed it only for the smallest models. The same open-weight model was reproducible or not depending only on how it was served. A pipeline we built to be deterministic at every stage -- text extraction and optical character recognition (OCR), chunking, embedding, indexing, retrieval and inference -- returned byte-identical output at each stage across reruns, interruptions, edits, machines of one graphics processing unit (GPU) model and days, for blood glucose meter and infusion pump documents. OCR, embedding and approximate retrieval varied when their own serving was not made invariant, and outputs changed with the GPU model and with the inference engine's scheduling settings, which therefore belong in the recorded configuration. End-to-end determinism is achievable and gives a testable basis for audit and change control.
| Comments: | 30 pages, 3 figures, 3 tables; supplementary tables S1-S15 and two supplementary notes included |
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.21300 [cs.CL] |
| (or arXiv:2610.21300v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21300 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ryosuke Suzuki [view email]
[v1]
Thu, 15 Oct 2026 05:45:20 UTC (125 KB)
来源：arXiv cs.CL · arxiv.org