arXiv cs.AI· Shengtao Wen, Yunying Yang, Xiang Chen, Lingbing Guo, Yu Tian, Sheng-Jun Huang·· 11 小时前AI 评分31
SPARK：面向 LLM 持续学习的任务后自蒸馏回放，解耦知识与隐私
Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning
AI 导读
SPARK 提出“保留-校正”分解框架，先冻结任务后学到的分布，再围绕该稳定参考做选择性校正，以在 LLM 持续学习中抑制敏感内容复现。其 Self-Distillation Replay 在学习当前任务时蒸馏旧任务行为，Post-Task Privacy Correction 降低标注 PII 的生成概率，同时将新旧任务的非 PII 行为锚定到该 checkpoint。
正文 · 原文
Abstract:Privacy-preserving continual learning (PPCL) must reduce the reproduction of sensitive content while retaining useful knowledge across sequential tasks. Formal privacy guarantees characterize randomized mechanisms, whereas operational output control concerns whether a trained model selectively reduces the likelihood of sensitive content in its outputs. In this work, we investigate the latter together with continual-learning utility under realistic task evolution. Retention and privacy correction operate at different granularities: task acquisition requires broad preservation of current- and old-task behavior, whereas privacy correction targets sparse annotated positions. Joint optimization leaves the current-task preservation target continually changing. We propose SPARK, a retention-correction decomposition that first freezes the learned post-task distribution and then applies selective correction around this stable reference. Self-Distillation Replay learns the current task while distilling behavior from previous tasks, and Post-Task Privacy Correction reduces annotated-PII likelihood while anchoring current- and old-task non-PII behavior to the resulting checkpoint. Extensive evaluations demonstrate that SPARK achieves effective selective PII suppression while preserving strong continual-learning utility and knowledge retention across diverse settings. Code and data will be released upon publication.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.29711 [cs.LG] |
| (or arXiv:2609.29711v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.29711 arXiv-issued DOI via DataCite |
Submission history
From: Shengtao Wen [view email]
[v1]
Mon, 31 Aug 2026 13:01:36 UTC (1,178 KB)
[v2]
Wed, 4 Nov 2026 08:23:54 UTC (1,938 KB)
来源：arXiv cs.AI · arxiv.org