跳到正文
arXiv cs.CL· Meilu Zhu, Xiaohan Li, Axiu Mao·· 9 小时前AI 评分36

HindAudit 审计临床智能体基准:后见之明从何进入模拟器与病例构建

Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction

AI 导读

HindAudit 将临床诊断智能体基准中的后见之明拆为三个渠道,在 MIMIC-IV 的 27,489 例急诊入院数据上审计发现,模拟化验结果未出现泄漏,给五个模拟器提供诊断的影响可忽略。

正文 · 原文

View PDF HTML (experimental)

Abstract:Benchmarks for LLM diagnostic agents are built from retrospective records: the case an agent reads is rewritten from discharge summaries, and unrecorded test results are generated by a simulator that often sees the final diagnosis. HindAudit separates three hindsight channels (the diagnosis given to the simulator, the retrospective context it conditions on, and the agent-facing case text) and measures each with judges trained only on real data. On 27,489 emergency admissions from MIMIC-IV, five registered tests found no leakage into simulated laboratory results: providing the diagnosis to five simulators had a negligible effect, and prompted gatekeepers diluted the diagnostic signal, although exploratory results suggest they can raise agent scores. The largest effect lay in the case text. Against a presentation-time reconstruction from triage records, benchmark-style summaries raised the zero-shot accuracy of four LLMs from 25.4% to 39.7% and discharge histories to 45.3%, and raised six interactive agents' scores while barely changing their ranking. The exploratory gap was reproduced in a registered replication on held-out patients. Masking diagnosis names left it unchanged. Because MIMIC lacks a contemporaneous bedside history, the gap is an upper bound on hindsight. We report all registered outcomes and recommend construction practices.
Comments: 8 pages main text, 20 pages in total, 6 figures, 11 tables
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2611.22241 [cs.CL]
  (or arXiv:2611.22241v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.22241

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Meilu Zhu [view email]
[v1] Tue, 17 Nov 2026 05:43:37 UTC (137 KB)

来源:arXiv cs.CL · arxiv.org