arXiv cs.CL· Christopher Kissling, Elena Merdjanovska, Alan Akbik·· 6 天前AI 评分22
语言模型关系知识置信度校准评估：Self-Aware Knowledge Probing 框架
Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration
AI 导读
研究者提出 Self-Aware Knowledge Probing 框架，在封闭答案集下评估语言模型关系知识的置信度校准，涵盖内在置信度、复述一致性和语言置信度三种模态。对 21 个因果语言模型和 8 个掩码语言模型的广泛分析显示，多数模型过度自信，尤以掩码目标预训练的模型为甚；考虑复述不一致性的置信度估计校准最佳，且增加答案选项数量可改善校准。
正文 · 原文
Abstract:Knowledge probing quantifies how much relational knowledge a language model (LM) has acquired during pre-training. Existing knowledge probes evaluate model capabilities through metrics like prediction accuracy and precision. Such evaluations fail to account for the model's reliability, reflected in the calibration of its confidence scores. In this paper, we establish an empirical framework to evaluate the calibration of relational knowledge in a closed answer-set setting. We cover three modalities of model confidence: (1) intrinsic confidence, (2) paraphrase consistency and (3) linguistic confidence. Our extensive analysis of 21 causal and 8 masked language models reveals that most models, but especially those pre-trained with the masking objective, are overconfident. The best-calibrated scores come from confidence estimates that account for inconsistencies due to statement rephrasing. Moreover, calibration improves when increasing the number of answer options.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2601.18901 [cs.CL] |
| (or arXiv:2601.18901v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.18901 arXiv-issued DOI via DataCite |
Submission history
From: Elena Merdjanovska [view email]
[v1]
Mon, 26 Jan 2026 19:11:48 UTC (224 KB)
[v2]
Thu, 8 Oct 2026 18:58:19 UTC (264 KB)
来源：arXiv cs.CL · arxiv.org