arXiv cs.CL· Ivan Kochergin, Vasily Konovalov·· 11 小时前AI 评分29
检索器越强，标签越失真：不完整相关性标注如何扭曲对话式 RAG 的拒答评估
Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAG
AI 导读
研究在 RECOR（RETECO track 2）上复现官方基线后发现，用“无 gold 文档即不可答”的捷径判定拒答会严重失真：检索器越强（nDCG@10 从 .18 到 .59），含答案的无 gold 上下文占比从 28% 升至 72%，该捷径精度从 .68 降至 .10（盲审下 .70 降至 .21）。
正文 · 原文
Abstract:Conversational RAG benchmarks such as MTRAG and RETECO evaluate retrieval and generation end to end: each system answers from its own retrieval, so abstention must be scored against whether each retrieved context is sufficient. A convenient shortcut reads sufficiency off the relevance judgments and treats a context without gold documents as unanswerable. We re-run the official baselines exactly and test this shortcut on RECOR (RETECO track~2) with four retrievers (nDCG@10 from .18 to .59) and three Qwen3 readers. Two open LLM judges and a panel of four small open models label each context by content, and a model of another family audits 250 contexts blind, with adjudication. The stronger the retriever, the larger the share of no-gold contexts that still contain the answer (28\% to 72\%) and the lower the shortcut's precision (.68 to .10; .70 to .21 under the audit). A reader agrees: Qwen3-4B answers 62--67\% of these false misses correctly, against 23--30\% of no-gold contexts judged insufficient and 1\% with no documents. In most false misses, the answer comes from a hard negative mined for the same question, another turn's gold or a neighbouring chunk of the gold page. On MTRAG, whose judgments come from annotators, precision drops from .72 to .50. Under the shortcut, answerability AUROC is up to .19 too low, and the best sufficiency detector appears to be a retrieval score that never reads the documents. We release code, runs and labels, and recommend scoring refusals against content labels, not the presence of gold.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.40966 [cs.CL] |
| (or arXiv:2610.40966v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40966 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vasily Konovalov [view email]
[v1]
Thu, 29 Oct 2026 20:52:24 UTC (126 KB)
来源：arXiv cs.CL · arxiv.org