arXiv cs.CL· Tobias Bersia, Tatiana Gaintseva·· 12 小时前AI 评分12
微调后的 Activation Oracles 为何学不会读取：概念特定的盲区
When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles
AI 导读
研究发现，在 Taboo Word Guessing 设定下微调的 Activation Oracles（AOs）会变成概念特定的"反读取器"，选择性地无法恢复其训练中持续存在的隐藏概念。
正文 · 原文
This paper has been withdrawn by Tatiana Gaintseva
No PDF available, click to view other formats
Abstract:Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.
| Comments: | A mistake was found in experimental results, so the results as presented in the current version of the paper are incorrect |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.23379 [cs.CL] |
| (or arXiv:2607.23379v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.23379 arXiv-issued DOI via DataCite |
Submission history
From: Tatiana Gaintseva [view email]
[v1]
Sat, 25 Jul 2026 21:58:17 UTC (2,172 KB)
[v2]
Thu, 5 Nov 2026 01:47:04 UTC (1 KB) (withdrawn)
来源：arXiv cs.CL · arxiv.org