跳到正文
arXiv cs.CL· Owen Cook, Xingyi Song·· 10 小时前AI 评分34

研究:LLM 已是具备置信度感知的零样本分类器

LLMs Are Already Confidence-Aware Zero-Shot Classifiers

AI 导读

研究对比黑盒与白盒置信度方法,在 11 个 1B-32B 开源 LLM 和 5 个分类数据集上评测发现,基于 logit 的打分方法 AUROC 达 0.74-0.77,明显优于黑盒方法的 0.57-0.66。

正文 · 原文

View PDF HTML (experimental)

Abstract:Zero-shot large language models (LLMs) are increasingly used as text classifiers, but their labels are only useful when we know to what extent we can trust them. We compare black-box methods (verbalised confidence and sampling) with white-box methods that read token probabilities (P(True), and multiple logit-scoring methods using constrained generation). We evaluate eleven open-weight LLMs (1B-32B parameters) on five classification datasets with 3 to 151 labels, measuring classification performance (accuracy, F1-macro), discrimination (AUROC), and calibration (ECE and Brier score). The choice of confidence method rarely impacts accuracy significantly, but logit-based scoring methods lead to much better discrimination (AUROC 0.74-0.77) than black-box methods (AUROC 0.57-0.66). Replacing labels with single-token identifiers preserves discrimination but costs 3-6 accuracy points on average. We also introduce decision-only scoring, which only scores positions where labels diverge in the prefix tree (trie). It reduces decode steps by 88-96% on the three largest label sets and performs comparably to full-label scoring. The best open-weight model-method configurations (Gemma 4 31B and Qwen 3.5 9B) are competitive with the commercial Jev 1.13.0 classifier.
Comments: 20 pages (9 pages main content), 2 figures. Code available at: this https URL
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2611.25289 [cs.CL]
  (or arXiv:2611.25289v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.25289

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Owen Cook [view email]
[v1] Wed, 18 Nov 2026 23:33:16 UTC (75 KB)

来源:arXiv cs.CL · arxiv.org