arXiv cs.CL· Ahmad Dawar Hakimi, Kaiwei Lei, Isabelle Augenstein, Hinrich Sch\"utze·· 13 小时前AI 评分31
用输出向量编辑抑制大语言模型记忆复现
Output Vector Editing for Memorization Mitigation in Large Language Models
AI 导读
研究者提出 Output Vector Editing（OVE）后处理方法，通过两阶段定位 MLP 层与神经元并做全词表最小二乘更新，抑制模型从给定前缀复现训练文本。在 634 对 OLMo-7B 数据上，其多锚点变体 OVE-MA 每次请求最多 5 次输出向量更新，可将可恢复比例降至 24.45%，优于 REVS 的 32.02%，更新操作约少 75 倍。该方法无需梯度，推理时也不需额外干预。
正文 · 原文
Abstract:Large language models can reproduce training text, creating privacy, copyright, and security risks. We propose Output Vector Editing (OVE), a post-hoc method for suppressing text continuations that a model reproduces from a given prefix. Under superposition, a neuron can contribute to multiple features. Yet some neuron-level mitigation methods suppress localized neurons by zeroing their activations, removing each neuron's entire contribution. OVE instead edits their output vectors, combining (i) two-stage localization of MLP layers and neurons with (ii) full-vocabulary least-squares updates. Its multi-anchor variant, OVE-MA, adaptively selects continuation positions expected to provide stronger suppression relative to the required weight change. We evaluate recovery and locality across four models (360M--7B parameters), using autoregressively validated continuations and recovery tests with greedy decoding, beam search, and sampling. On 634 OLMo-7B pairs, OVE-MA with at most five output-vector updates per request leaves 24.45\% recoverable in these tests, compared with 32.02\% for the closest prior post-hoc editing method, REVS, while using approximately 75 times fewer update operations. These results show that adaptive position selection can concentrate effective suppression into a small number of weight updates. OVE edits weights without gradients and requires no additional intervention during inference.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.18767 [cs.CL] |
| (or arXiv:2606.18767v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.18767 arXiv-issued DOI via DataCite |
Submission history
From: Ahmad Dawar Hakimi [view email]
[v1]
Wed, 17 Jun 2026 07:29:18 UTC (1,299 KB)
[v2]
Wed, 21 Oct 2026 13:21:33 UTC (1,031 KB)
来源：arXiv cs.CL · arxiv.org