arXiv cs.AI· Damiano Fornasiere, Mirko Bronzi, Spencer Kitts, Alessandro Palmas, Yoshua Bengio, Oliver Richardson·· 2 天前AI 评分40
语言模型能识别自身激活上的 dropout 与高斯噪声
Language models recognize dropout and Gaussian noise applied to their activations
AI 导读
研究发现语言模型能检测、定位并一定程度描述施加在其激活上的语义无关扰动。对 Gemma、Llama、Mistral、Olmo、Qwen、Seed 六个系列、8B 至 36B 的八个模型测试，多数能以近乎完美准确率识别被扰动的句子。Olmo 在预训练中形成该能力，Qwen3-32B 与 Qwen3.8-27B 识别扰动类型的零样本准确率与扰动强度相关。
正文 · 原文
Abstract:We provide evidence that language models can detect, localize and, to a degree, verbalize the difference between a-semantic perturbations applied to their activations. More precisely, we either (a) apply dropout or (b) add Gaussian noise to them, at a target sentence. We then ask a multiple-choice question such as "Which of the previous sentences was perturbed?" or "Which perturbation was applied?". We test eight models from the Gemma, Llama, Mistral, Olmo, Qwen and Seed families, with sizes between 8B and 36B, all of which can easily detect and localize the perturbations, often with perfect accuracy. To investigate the mechanisms, we find that Olmo models develop this capacity during pre-training, and we rule out the hypothesis that perturbed sentences simply have lower log-likelihood. Notably, the zero-shot accuracy of Qwen3-32B and Qwen3.8-27B in identifying which perturbation was applied correlates with the perturbation strength. While this task proves hard for the remaining models, most of them can learn, when taught in context, to distinguish between dropout and Gaussian noise. Interestingly, Qwen3-32B and Qwen3.8-27B learn worse when supervised with wrong labels, again suggesting a prior towards the correct ones. Because dropout is used as a training-regularization technique, while Gaussian noise is sometimes added during inference, we discuss the possibility of a data-agnostic "training awareness" signal and the implications for AI safety.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.17465 [cs.AI] |
| (or arXiv:2604.17465v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.17465 arXiv-issued DOI via DataCite |
Submission history
From: Damiano Fornasiere [view email]
[v1]
Sun, 19 Apr 2026 14:30:13 UTC (94 KB)
[v2]
Thu, 30 Apr 2026 22:17:02 UTC (94 KB)
[v3]
Wed, 14 Oct 2026 19:43:24 UTC (6,905 KB)
来源：arXiv cs.AI · arxiv.org