arXiv cs.CL· Jingkai Guo, Ziyi Zhao, Chaitali Chakrabarti, Deliang Fan·· 12 小时前AI 评分38
CoLU：面向大语言模型的置信度加权分层遗忘方法
CoLU: Confidence-Weighted Layerwise Unlearning for Large Language Models
AI 导读
研究者提出置信度加权分层遗忘方法 CoLU，只更新单个选定的 Transformer 层而非整个模型，并配合新的置信度加权遗忘与保留正则化。配套的 Brief Layer Screening（BLS）用归一化遗忘-保留（NFR）分数预筛层候选，将 CoLU 运行时间降低 60.2-79.7% 且性能几乎不变。
正文 · 原文
Abstract:Large language model (LLM) unlearning aims to forget certain information while preserving others, without retraining from scratch. SoTA methods such as NPO, SimNPO, and SatImp balance forgetting and retain through whole model fine-tuning, while FOM-UL explores selective layer updates. Prior works show that suppressing target information can damage retained knowledge. This trade-off can vary across layers even under the same unlearning loss. In this work, we propose Confidence-Weighted Layerwise Unlearning (CoLU), which combines a novel confidence-weighted forgetting and retain regularization while updating a single selected transformer layer, rather than the whole model for unlearning. To reduce CoLU compute cost, we further propose Brief Layer Screening (BLS), which uses a newly defined normalized forgetting-retention (NFR) score to first rank layers through a brief unlearning trials using a subset of forgetting dataset and fewer epochs. BLS pre-selects top-k ranked layer candidates, which will then be fully profiled using full data analysis. BLS is proven to reduce the runtime of CoLU by 60.2-79.7% with almost the same performance. On TOFU dataset, CoLU improves both forgetting and retention over NPO, SatImp, and GA-based FOM-UL in all six TOFU settings, and over SimNPO in five of the six settings. On MUSE Books and News datasets, CoLU achieves the highest retained KnowMem and the PrivLeak closest to zero among the compared SOTA unlearning methods. CoLU presents a new record performance in both unlearning and retain in most settings up to date. Moreover, across the measured settings, average CoLU GPU memory usage is 77.4% to 79.5% lower than full-model unlearning.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.22721 [cs.CL] |
| (or arXiv:2610.22721v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22721 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jingkai Guo [view email]
[v1]
Thu, 15 Oct 2026 22:51:35 UTC (117 KB)
来源：arXiv cs.CL · arxiv.org