arXiv cs.CL· Heyuan Huang, Jirui Dai, Alexandra DeLucia, Sonal Joshi, Mahsa Yarmohammadi, Jie Gao, Bernal Jim\'enez Guti\'errez, Mark Dredze·· 12 小时前AI 评分35
基于检索的开放式评估在哪里失效？从长篇医学答案事实性验证自动归纳失败分类体系
Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification
AI 导读
研究针对基于检索的医学事实性评估，从MedExpert开放式数据集与3个封闭式数据集归纳出两套失败分类体系：检索阶段按五个质量维度分解错误，验证器推理错误拆为六个连续步骤。团队用LLM-as-Judge自动标注证据质量并在4种检索方法与6个前沿验证模型上压力测试，发现扩大模型规模、增加推理投入、引入权威网络来源与医学微调均无法消除这些失败模式。
正文 · 原文
Abstract:Retrieval-based factuality evaluation, where LLM-generated claims are verified against evidence from authoritative medical corpora, has become the dominant paradigm for scalable hallucination detection in high-stakes clinical settings. Despite the urgency of reliable and transparent medical fact verification, most systems measure performance with aggregate metrics like F1, which obscure where and why failures occur. Existing RAG diagnostics require gold answers or annotated gold evidence, neither of which exists in this regime. We introduce two comprehensive taxonomies, grounded in a case study on the open-ended MedExpert dataset and 3 closed-ended datasets, decomposing failures into retrieval-stage errors along five quality dimensions, and verifier-reasoning errors into six consecutive steps. We adapt an automatic pattern induction pipeline using LLM-as-Judge to label evidence quality and classify verifier reasoning errors at scale, and then stress-test our findings across 4 retrieval methods and 6 frontier verifier models. Our analysis reveals that scaling model size, adding reasoning effort, expanding to authoritative web sources, and applying medical fine-tuning do not resolve these failure modes, demonstrating that they represent fundamental limitations of the retrieve-then-verify paradigm in open-ended medical settings rather than artifacts of outdated systems. We release our code and data at this https URL for the full reproducibility of our results.
| Comments: | All experiments were done before May 2026 (also all corpus knowledge cutoff date). Accepted as an Oral paper (Best Paper Finalist) at NeurIPS 2026 Generative AI for Health |
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2609.30467 [cs.CL] |
| (or arXiv:2609.30467v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.30467 arXiv-issued DOI via DataCite |
Submission history
From: Heyuan Huang [view email]
[v1]
Thu, 24 Sep 2026 19:04:23 UTC (349 KB)
[v2]
Thu, 5 Nov 2026 17:22:33 UTC (2,699 KB)
来源：arXiv cs.CL · arxiv.org