arXiv cs.CL· Parv Khatri, Harshul Jain, Mehak Kapoor, Tanya Sah·· 2 天前AI 评分34
判断 RAG 知识冲突类型，真的需要 LLM 吗？决策模型分类研究
Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG
AI 导读
研究用单次前向传播输出固定选项概率的决策模型，在 CONFLICTS 基准上对 RAG 知识冲突类型分类，11 个模型准确率 50.0–67.0%，其中 10 个达到或超过 Gemini 2.5 Flash 的 61.3%，计费成本低 4–39 倍、延迟低 2.6–6.0 倍。
正文 · 原文
Abstract:When the sources retrieved for a query disagree, the right response depends on why they disagree: a question with several valid answers, a contested topic, stale information and a false source each call for different behaviour. Cattan et al. (2025) showed that telling a generator the conflict type improves its responses by about 24 points, but that predicting the type is hard even for frontier LLMs. We study whether typed decision models, which return probabilities over fixed options in a single forward pass instead of generating text, can do this classification fast and cheaply. Using the CONFLICTS benchmark and the original paper's prompt verbatim, we evaluate all 11 decision models served by one commercial inference gateway, from 7 providers, against Gemini models on identical inputs. Asked directly, the decision models reach 50.0--67.0% accuracy; 10 of the 11 match or exceed Gemini 2.5 Flash, the paper's model (61.3%), at 4--39x lower billed cost (2 were free) and 2.6--6.0x lower latency. The current Gemini 3.8 Flash (70.0%) is significantly more accurate than 8 of them, but sending each decision model's least confident quarter of queries to it gives a cascade statistically tied with it for 8 of the 11, at 25--37% of its cost. End to end, answers generated with the types of 10 of the 11 follow the expected behaviour as often as with the LLM's types. Decomposing the taxonomy into yes/no questions lowers accuracy for every model, and a repair fitted on one model does not transfer. Pre-specified diagnostics show that dates play no measurable role and that sensitivity to source order varies widely between models. Decision models detect a planted false source less reliably than the LLM as hard labels, but their probabilities still separate planted from clean sources. We release code, prompts and all model outputs at this https URL.
| Comments: | 15 pages, 2 figures, 8 tables. Code, prompts and all model outputs: this https URL |
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.36915 [cs.CL] |
| (or arXiv:2610.36915v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36915 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Parv Khatri [view email]
[v1]
Tue, 27 Oct 2026 09:43:21 UTC (93 KB)
来源：arXiv cs.CL · arxiv.org