arXiv cs.CL· Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias, Tha\'is Gaud\^encio, Yuri Malheiros, Barufaldi·· 13 小时前AI 评分20
标注覆盖如何影响检索分数：乳腺病变 RAG 系统的证据感知评估
How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions
AI 导读
一项针对乳腺病变管理文献的临床 RAG 系统研究提出证据感知检索评估协议，指出检索分数依赖标注覆盖度。
正文 · 原文
Abstract:Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.
| Comments: | 18 pages, 3 figures, 6 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.31606 [cs.CL] |
| (or arXiv:2610.31606v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31606 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Miguel Mochizuki Silva [view email]
[v1]
Thu, 22 Oct 2026 17:36:14 UTC (344 KB)
来源：arXiv cs.CL · arxiv.org