答案评估如何误导置信度选择：面向开放式问答的 FactForm 事实锚定表单评分器

View PDF HTML (experimental)

Abstract:Many benchmarks now prefer open-ended questions for testing LLM and NLP methods. We find that the choice of a calibration rule can flip when only the correctness labels change: on 141 pilot answers, rules chosen with exact-match labels look 0.239 Brier points better than raw confidence but are 0.103 worse under human labels. How such flips arise remains unclear. We look deeply into their mechanism: reference matching rejects many correct answers (exact match misses 42%), and an exact Brier identity shows these errors favour rules giving such answers lower confidence, which abstain on them at common thresholds. Guided by this, we propose FactForm, a grader that trusts reference matches and otherwise consults evidence retrieved from the question alone. One LLM call per answer fills a fixed-order closed form that a program scores. On 998 fresh EVOUNA answers with 400 fact-based human labels, FactForm reaches a balanced error of 0.077 and an AUROC of 0.985, against 0.112 and 0.930 for the best of eight traditional graders. Its same-model stability is no worse than that of LLM judges (mean absolute run-to-run difference 0.0136 against 0.0159 to 0.0206). We hope this helps benchmarks grade open-ended answers more accurately, with repeatable and discriminative scores.

Comments:	v2: substantially revised and extended. The study now covers open-ended question answering instead of theory-of-mind tracking, and adds the FactForm grader and a confirmation study on 998 fresh answers. 20 pages, 4 figures, 11 tables
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
ACM classes:	I.2.7; I.2.6
Cite as:	arXiv:2608.25654 [cs.CL]
	(or arXiv:2608.25654v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2608.25654 arXiv-issued DOI via DataCite

Submission history

From: Zhexi Feng [view email]
[v1] Wed, 26 Aug 2026 11:36:31 UTC (1,166 KB)
[v2] Mon, 2 Nov 2026 00:28:08 UTC (467 KB)