arXiv cs.AI· Aashiq Muhamed, Jacopo Bonato, Mona Diab, Virginia Smith·· 3 天前AI 评分27
SAEs 能否改进 LLM 遗忘：动态稀疏自编码器护栏实现精准遗忘
SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs
AI 导读
研究提出 Dynamic DAE Guardrails（DSG），一种利用原则性特征选择与动态分类器的精准遗忘方法，证明稀疏自编码器（SAEs）在动态使用时能显著提升 LLM 遗忘效果，打破此前 SAEs 不如梯度方法的结论。DSG 在遗忘-效用权衡上优于主流方法，并提升计算效率与稳定性、支持序列遗忘、抵抗重学习攻击、改善数据效率（含零样本）且更具可解释性。
正文 · 原文
Abstract:Machine unlearning is a promising approach to improve LLM safety by removing unwanted knowledge from the model. However, prevailing gradient-based unlearning methods suffer from issues such as high computational costs, hyperparameter instability, poor sequential unlearning capability, vulnerability to relearning attacks, low data efficiency, and lack of interpretability. While Sparse Autoencoders are well-suited to improve these aspects by enabling targeted activation-based unlearning, prior approaches underperform gradient-based methods. This work demonstrates that, contrary to these earlier findings, SAEs can significantly improve unlearning when employed dynamically. We introduce $\textbf{Dynamic DAE Guardrails}$ (DSG), a novel method for precision unlearning that leverages principled feature selection and a dynamic classifier. Our experiments show DSG substantially outperforms leading unlearning methods, achieving superior forget-utility trade-offs. DSG addresses key drawbacks of gradient-based approaches for unlearning -- offering enhanced computational efficiency and stability, robust performance in sequential unlearning, stronger resistance to relearning attacks, better data efficiency including zero-shot settings, and more interpretable unlearning.
| Comments: | COLM 2025 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2504.08192 [cs.LG] |
| (or arXiv:2504.08192v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2504.08192 arXiv-issued DOI via DataCite |
Submission history
From: Aashiq Muhamed [view email]
[v1]
Fri, 11 Apr 2025 01:24:03 UTC (1,413 KB)
[v2]
Mon, 7 Sep 2026 03:32:59 UTC (1,409 KB)
[v3]
Sat, 10 Oct 2026 00:07:02 UTC (1,406 KB)
来源：arXiv cs.AI · arxiv.org