arXiv cs.CL· Haozheng Liu, Zhaoqian Xue, Zhongyin Wang, Junting Mu, Zihui Ma, Shuya Feng, Lingyao Li·· 13 小时前AI 评分34
决策模型能否替代 LLM 做幻觉检测评判？arXiv 论文对比 gpt-6-sol
Decision Models as Judges for Hallucination Detection
AI 导读
研究构建了覆盖五类幻觉的二元基准（6 个数据集共 9,852 条），对比五种决策模型与生成式评判模型 gpt-6-sol。最佳决策模型在上下文不一致上与 gpt-6-sol 的平衡准确率持平，事实错误上差距在 3 个百分点内，但逻辑与指令不一致上落后 4-8 个百分点。
正文 · 原文
Abstract:Large language models are widely used as judges of model outputs (LLM-as-a-Judge), but they are costly and can hallucinate or be overconfident. Decision models are a potential alternative. They return a predefined verdict with a probability for each label instead of free-form text and are much cheaper and faster. We test this promise on hallucination detection. We build binary benchmarks for five hallucination types (9,852 items from six datasets) and compare five decision models with the generative judge gpt-6-sol. Performance depends strongly on the type: the best decision model matches gpt-6-sol in balanced accuracy on context inconsistency and comes within 3 percentage points on factual errors, but trails it by 4-8 points on logical and instruction inconsistency. Decision-model probabilities are overconfident, though less so than the GPT-6 models, and separate correct from incorrect decisions only moderately. We also find that a confidence cascade of decision models improves on the best single model by 1.9-3.3 points on three types and matches or exceeds gpt-6-sol at 1-3% of its cost. Overall, decision models are a practical low-cost judge for some hallucination types but need to be validated before use.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.37229 [cs.CL] |
| (or arXiv:2610.37229v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37229 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haozheng Liu [view email]
[v1]
Tue, 27 Oct 2026 13:38:28 UTC (401 KB)
来源：arXiv cs.CL · arxiv.org