对前沿 LLM 知识 KB 抽取的审计：GPTKB v1.5 的多维分析

View PDF HTML (experimental)

Abstract:LLMs are remarkable artifacts that have revolutionized a range of knowledge-intensive tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we provide a framework and the results of a multi-dimensional analysis of GPTKB v1.5 (Hu et al., 2025a), a recursively elicited Knowledge Base (KB) of 100 million facts (or beliefs) of a frontier LLM, namely, GPT-4.1. Given the scale of the elicited facts, we provide a multi-dimensional approach to qualitatively and quantitatively analyze these facts as opposed to the mainstream fact completion benchmarks, which are prone to availability bias. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities in neuro-symbolic AI concerning extraction, consolidation and verification of factual LLM knowledge.

Comments:	Accepted at AKBC@EMNLP 2026
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2510.07024 [cs.CL]
	(or arXiv:2510.07024v4 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2510.07024 arXiv-issued DOI via DataCite

Submission history

From: Shrestha Ghosh [view email]
[v1] Wed, 8 Oct 2025 13:48:38 UTC (114 KB)
[v2] Thu, 9 Oct 2025 07:23:03 UTC (114 KB)
[v3] Fri, 18 Sep 2026 13:29:31 UTC (105 KB)
[v4] Tue, 13 Oct 2026 13:17:59 UTC (105 KB)