arXiv cs.CL· Federico Torrielli, Peter Schneider-Kamp, Lukas Galke Poech·· 13 小时前AI 评分26
激活预言机的置信度与校准：让语言模型内部表征解读更可靠
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
AI 导读
研究对比了五种估计激活预言机置信度的方法，覆盖 Qwen 和 Gemma 两个模型家族、8B 至 27B 参数的四个预言机，在秘密单词任务上每种组合测试 6000 个样本。
正文 · 原文
Abstract:An activation oracle is a language model trained to convert the activations of another model into natural language. However, it is so far unclear to what extent these verbalizations are calibrated. Crucially, the oracle's verbalization come without any measure of confidence. Here, we compare five methods for estimating the confidence of an activation oracle. Our experiments cover four oracles from two model families (Qwen and Gemma, from 8B to 27B parameters), tested on the secret word task, whereby the oracle is supposed to recover a secret word, that has been fine-tuned into a model but does not appear in the model outputs. Our results with $6{,}000$ samples per method/oracle combination show that the best calibration depends on the availability of a hypothesis space or a calibration set: When an assumed auditor can list the possible answers in advance, the oracle should score each candidate answer directly. Scoring the candidates gives a higher accuracy on every oracle and a confidence that separates correct from incorrect answers better than any other method. When the oracle has to write its answer as free text, the agreement over twenty sampled answers gives the best calibrated confidence, which needs only five labeled secret words to be tuned. When a calibration set is available, a rescaled probability of the answer is as good as re-sampling, at the cost of one generation instead of twenty. Asking the oracle to write a number for its confidence, on the other side, gives no useful signal on any oracle. Code and the patched trainer are available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.26045 [cs.CL] |
| (or arXiv:2605.26045v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.26045 arXiv-issued DOI via DataCite |
Submission history
From: Federico Torrielli [view email]
[v1]
Mon, 25 May 2026 17:08:47 UTC (1,381 KB)
[v2]
Mon, 3 Aug 2026 15:43:52 UTC (365 KB)
[v3]
Thu, 22 Oct 2026 12:57:10 UTC (3,487 KB)
来源：arXiv cs.CL · arxiv.org