FIRM：面向联邦 LLM 遗忘的 Fisher 感知保留合并框架

View PDF HTML (experimental)

Abstract:Federated LLM unlearning addresses targeted knowledge removal when forget and retain data remain distributed across private clients and are inaccessible to the server. The resulting locally trained adapters encode conflicting objectives, making their aggregation fundamentally different from conventional federated optimization. A key failure mode is magnitude explosion, whereby unnormalized cross-client aggregation increases the merged update magnitude and degrades retained utility under stronger unlearning. We propose FIRM (Fisher-Informed Retention Merging), a single-round framework for stable and retention-aware server-side aggregation. FIRM uses client-local Fisher statistics as a unified sensitivity signal to construct a Fisher-normalized base merge and impose explicit control over retain-sensitive parameters. Its second-order retention projection admits a unique minimum-distance solution, while the corresponding dual scales with the number of retain clients rather than the model dimension. Experiments across WMDP, TOFU, and MUSE-HP using Zephyr-7B and Llama-3-8B show utility gains at comparable forgetting on WMDP and MUSE-HP, with metric-dependent trade-offs on TOFU. Controlled comparisons support benefits of the combined base merge and additional retention correction at selected operating points.

Comments:	16 pages, 1 figure
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:	arXiv:2610.21720 [cs.LG]
	(or arXiv:2610.21720v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.21720 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Reeyansh Shah [view email]
[v1] Thu, 15 Oct 2026 11:39:55 UTC (2,176 KB)