arXiv cs.CL· Atharva Thakur, Abulhair Saparov, Bruno Ribeiro·· 12 小时前AI 评分31
DeclineCraft：联合衡量检索与参数知识的 RAG 弃答基准
DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge
AI 导读
研究者提出 DeclineCraft，一个以 Minecraft 在语料截止后新增功能为不可答问题的动态弃答基准，并审计模型的参数知识暴露情况。在 27 个 LLM 上测试发现，单次检索在 40 种配置中的 38 种显著提升正确弃答率（CAR），而迭代 RAG 在 30 种配置中显著降低 CAR（均值 46.4%），答案准确率仅从 68.5% 变为 69.3%。
正文 · 原文
Abstract:Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer. We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure. Later cut-offs support evaluation of future models. Across 27 LLMs, we measure Correct Abstention Rate (CAR; how often models acknowledge insufficient knowledge on unanswerable questions), Answerable Attempt Rate and Answer Accuracy. We find that (1) single-shot retrieval significantly raises CAR in 38 of 40 configurations (model and thinking setting) versus no retrieval. (2) Iterative RAG, in which the model self-assesses its context and can request up to two further retrieval rounds, significantly lowers CAR versus single-shot RAG in 30 of 40 configurations (mean 46.4%), with little change in Answer Accuracy (68.5% to 69.3%). Adding query rewriting and reranking to hybrid (dense & sparse) retrieval lowers CAR for all 12 models tested. (3) Larger models answer more accurately (Spearman $\rho = +0.60$ with total parameters), but neither model size nor release date shows a significant association with CAR. (4) Without retrieval, enabling thinking on the same weights lowers CAR significantly for 8 of 13 models (by 14% points on average), and the responses shift from acknowledged uncertainty to premise denial.
| Comments: | 38 pages, 28 figures, 12 tables. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.28340 [cs.CL] |
| (or arXiv:2610.28340v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28340 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Atharva Thakur [view email]
[v1]
Tue, 20 Oct 2026 15:07:19 UTC (709 KB)
来源：arXiv cs.CL · arxiv.org