arXiv cs.AI· Fei Wang, Baochun Li·· 1 天前AI 评分38
更省训练、更少泄露：重新审视 LoRA 微调中的 LLM 记忆化
Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA
AI 导读
研究重新审视了 LoRA 微调中的记忆化问题，发现模型规模和数据重复等在前训练与全量微调中显著影响记忆化的因素，在 LoRA 微调中并不遵循相同趋势。采用更宽松的基于相似度的记忆化指标，LoRA 相比全量微调显著降低了记忆化风险，同时仍保持较强的任务性能。
正文 · 原文
Abstract:Memorization in large language models (LLMs) makes them vulnerable to data extraction attacks. While pre-training memorization has been extensively studied, fewer works have explored its impact in fine-tuning, particularly for LoRA fine-tuning, a widely adopted parameter-efficient method.
In this work, we re-examine memorization in fine-tuning and uncover a surprising divergence from prior findings across different fine-tuning strategies. Factors such as model scale and data duplication, which strongly influence memorization in pre-training and full fine-tuning, do not follow the same trend in LoRA fine-tuning. Using a more relaxed similarity-based memorization metric, we demonstrate that LoRA significantly reduces memorization risks compared to full fine-tuning, while still maintaining strong task performance.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2506.20856 [cs.LG] |
| (or arXiv:2506.20856v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.20856 arXiv-issued DOI via DataCite |
Submission history
From: Fei Wang [view email]
[v1]
Wed, 25 Jun 2025 22:01:25 UTC (281 KB)
来源：arXiv cs.AI · arxiv.org