arXiv cs.CL· Hanyu Qiu, Ze Liu, Zhe Zhao, Kuang Xiong, Blaze Chen, Zhenxiang Yan, Renfen Hu·· 11 小时前AI 评分38
HaluCheck：评估 LLM 能否可靠检测文本与智能体轨迹中的多维幻觉
From Twisted Facts to Rogue Agents: Can LLMs Reliably Detect Multi-Dimensional Hallucinations?
AI 导读
研究者推出双语基准 HaluCheck，含超 42k 条实例，覆盖文本事实幻觉与智能体轨迹忠实性幻觉，并提供检测、定位、纠正的多级细粒度标注。对 14 个 LLM 的实验显示，模型仅具备样本级基础检测能力，在细粒度定位、纠正和智能体幻觉检测上表现不佳。思维链、网页搜索与监督微调各带来部分改善，但均未彻底解决问题。
正文 · 原文
Abstract:Large language models (LLMs) are widely used for information access, content production, and agent-based task execution, but their outputs remain vulnerable to subtle, high-risk hallucinations. To systematically evaluate and advance post-generation hallucination detection, we introduce HaluCheck, a large-scale, bilingual benchmark comprising over 42k instances spanning both factual hallucinations in text and faithfulness hallucinations in agent trajectories. HaluCheck features fine-grained, multi-level annotations for detection, localization, and correction, covering different tasks and perturbation degrees. Experiments on fourteen LLMs reveal that, while models show basic sample-level detection ability, they struggle with fine-grained localization, correction, and agent hallucination detection. Further analysis demonstrates that thinking, web search, and supervised fine-tuning each offer partial improvements, but none fully solve the problem. HaluCheck thus provides a solid basis for benchmarking and diagnosing LLM reliability across text and agent scenarios.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.08231 [cs.CL] |
| (or arXiv:2611.08231v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08231 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanyu Qiu [view email]
[v1]
Thu, 5 Nov 2026 16:15:18 UTC (2,087 KB)
来源：arXiv cs.CL · arxiv.org