arXiv cs.AI· Hankyeol Kim, Pilsung Kang·· 12 小时前AI 评分31
同一答案，不同置信度：LLM 置信度校准中的协议敏感性
Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibration
AI 导读
在五个开源 LLM 和四个问答数据集上，固定答案、口头置信度与正确性标签、仅改变测量协议，token 置信度与口头置信度的校准对比结果会随之改变。
正文 · 原文
Abstract:Large language models (LLM) can state their confidence in an answer (verbalized confidence), or the confidence can be read from the probability of the answer's tokens (token confidence). Comparisons of the two ask which is better calibrated, but the token probability of an answer depends on the text that precedes it. Across five open LLMs and four question-answering datasets, we fix each answer, its verbalized confidence, and its correctness label, and vary how each score is measured. The scores themselves change with the protocol, and so does the comparison between them. Changing the context in which the token probability is read shifts the gap between the two scores by about the size of the gap itself (.287 vs. .304 in expected calibration error, ECE; .189 vs. .145 in AUROC). The 95% interval of the shift excludes zero in 19 and 17 of the 20 settings, and under AUROC the shift has the same sign in all of them, so the context changes how answers are ranked. The better score flips in 8 settings under each metric, counting only flips that the intervals support, and the shift also appears in four larger models of up to 72B parameters. Most of the shift appears with changes to the instruction, not to the position of the answer. Other measurement choices also matter, and stated confidence depends on how a supplied answer is written. We turn these results into a reporting checklist for LLM confidence and its comparisons.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.27752 [cs.AI] |
| (or arXiv:2605.27752v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.27752 arXiv-issued DOI via DataCite |
Submission history
From: Hankyeol Kim [view email]
[v1]
Tue, 26 May 2026 23:03:38 UTC (60 KB)
[v2]
Mon, 1 Jun 2026 13:50:34 UTC (60 KB)
[v3]
Fri, 7 Aug 2026 08:03:29 UTC (437 KB)
[v4]
Fri, 23 Oct 2026 14:58:08 UTC (453 KB)
来源：arXiv cs.AI · arxiv.org