arXiv cs.CL· Dongyao Chen, Jiayun Lei, Zhiyuan Ji, Zhiying Deng, Wei Liu, Xiaobo Sun·· 10 小时前AI 评分24
从源头分配容量：面向 LLM 知识编辑的稀疏目标优化
Capacity Allocation at the Source: Sparse Target Optimization for LLM Knowledge Editing
AI 导读
研究提出一种 ℓ1 正则化方法，通过剪除知识向量中多余的“触须”、仅保留必要分量，减少 LLM 知识编辑时对其他知识的干扰。作者从理论上证明参数空间干扰与目标隐藏状态之间的干扰正相关，并在多个 LLM 和基准数据集上验证了该方法的效果。代码已公开。
正文 · 原文
Abstract:Knowledge editing has emerged as a promising approach for efficiently updating embedded knowledge in large language models (LLMs). It first computes an ideal target hidden state that steers the LLM toward the new output, and then edits the model parameters so that the original hidden state is mapped to this target. While most existing methods focus on the second stage of parameter editing in preserving other knowledge, we find that the upstream construction of the target hidden state also plays an important role in allocating the model's limited capacity space. We first establish a theoretical connection between interference in the parameter space and interference between target hidden states, showing that the two are positively correlated. Building on this insight, we propose a simple yet effective $\ell_1$ regularization method that prunes away unnecessary ``tentacles'' of each knowledge vector, retaining only the essential components. With fewer extraneous tentacles, each edit affects less other knowledge, enabling better coexistence. Extensive experiments across multiple LLMs and benchmark datasets verify the effectiveness of our method. Code: this https URL.
| Comments: | neurips2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.12995 [cs.CL] |
| (or arXiv:2611.12995v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.12995 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lei Jiayun [view email]
[v1]
Sun, 8 Nov 2026 13:28:43 UTC (260 KB)
来源：arXiv cs.CL · arxiv.org