arXiv cs.CL· Davide Baldelli, Robyn Kozierok, Mirko Bronzi, Damiano Fornasiere·· 3 天前AI 评分36
跨模型上下文神经反馈实验揭示 LLM 的（有限）激活控制能力
Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs
AI 导读
研究者用上下文神经反馈实验测试十个开源模型（来自五个模型族，规模 9B 至 32B）能否控制自身激活值。在固定句子预填充条件下，十个模型中有三个表现出小幅但一致的激活控制效果；在欺骗监测器测试中，没有模型学会降低监测分数，但有一个模型能将其提高。作者认为，模型对自身激活的上下文控制目前尚不构成对激活监测器的威胁，但已在少数模型中出现，需随模型演进持续追踪。
正文 · 原文
Abstract:Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.
| Comments: | 29 pages, 11 figures, 5 tables. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20179 [cs.CL] |
| (or arXiv:2610.20179v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20179 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Davide Baldelli [view email]
[v1]
Wed, 14 Oct 2026 13:24:26 UTC (512 KB)
来源：arXiv cs.CL · arxiv.org