为 LLM 建议引入“认知担保”：在缺乏真实答案时评估建议的可依赖范围

View PDF HTML (experimental)

Abstract:Large language models are increasingly used to support organizational decisions. Despite their rapid and widespread adoption, users often lack a principled basis for assessing whether to rely on a specific recommendation. We introduce \emph{epistemic warrant}, a decision-level construct that characterizes the scope over which a recommendation is supported. We operationalize this construct through a reliance certificate that uses behavioral evidence from four tiers of tests to assess a recommendation's underlying warrant and assign it to a warrant category. We assess this operationalization using complementary forms of construct-validity evidence, including transformation validity, known-groups tests, independent human consensus, discriminant analyses, and robustness across models and alternative implementations. The results show that the certificate behaves in theoretically expected ways: its transformations instantiate their intended relationships, certificate assignments increase across groups prespecified to represent increasing levels of warrant, and stronger assigned warrant is associated with greater independent human consensus. Certificate assignments also capture information beyond verbalized confidence and are not readily explained by decision difficulty. The main conclusions remain robust across models and operationalizations. These findings support the reliance certificate as an operationalization of epistemic warrant for assessing the scope of support for a particular LLM recommendation when ground truth is unavailable.

Submission history

From: Shai Vardi [view email]
[v1] Thu, 3 Sep 2026 17:25:20 UTC (253 KB)
[v2] Thu, 29 Oct 2026 13:10:33 UTC (267 KB)