ProbeDrift：面向可靠 OOD 不确定性量化的探针鲁棒性系统评估

View PDF

Abstract:Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation, motivating a growing interest in efficient probe-based approaches. Yet it remains unclear how robust existing methods are, with prior work reporting conflicting conclusions under substantially different evaluation settings. We address this by introducing ProbeDrift, a systematic evaluation framework for supervised uncertainty probes covering a wide range of OOD settings across models, tasks, and distributional shifts. Using ProbeDrift, we train over 2,000 probes to disentangle the effect of key design choices, showing poor robustness of current methods beyond near-OOD settings. We find that robustness is driven by design decisions that have a largely invisible effect in-distribution, including the choice of feature type, aggregation strategy, and training signal. We argue that robust uncertainty estimation requires robust evaluation. To support this, we release ProbeDrift as a lightweight Python library that contains the train and test splits underpinning our extensive evaluation. We also show how insights from our evaluation can directly lead to more robust methods through a simple Hybrid Back-Off (HBO) strategy.

Comments:	Accepted at EMNLP 2026
Subjects:	Computation and Language (cs.CL)
ACM classes:	I.2.7
Cite as:	arXiv:2604.11662 [cs.CL]
	(or arXiv:2604.11662v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2604.11662 arXiv-issued DOI via DataCite

Submission history

From: Joe Stacey [view email]
[v1] Mon, 13 Apr 2026 16:11:33 UTC (1,195 KB)
[v2] Sun, 30 Aug 2026 16:23:41 UTC (1,210 KB)
[v3] Wed, 21 Oct 2026 10:45:47 UTC (1,210 KB)