arXiv cs.AI· Zhilong Zheng, Letian Tao, Yujie Yang, Wei Xiong, Kehua Sheng, Bo Zhang, Jingliang Duan, Keqiang Li, Yang Guan, Shengbo Eben Li·· 14 小时前AI 评分31
JANUS：用事后权重校正缓解微调中的稳定性-可塑性困境
Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification
AI 导读
研究者提出 JANUS，一个纯事后、与微调方法无关的权重校正框架，通过将参数更新投影到 Jacobian 零空间（JANUS）实现参数空间正交，从而在不干扰微调过程的前提下恢复历史知识。
正文 · 原文
Abstract:Fine-tuning foundation models on new tasks inevitably suffer from catastrophic forgetting. While existing works attempt to mitigate this on the basis of parameter-efficient fine-tuning methods, they adopted an overly restrictive Subspace Orthogonality condition. In this paper, we introduce a purely post-hoc and tuning-agnostic weight rectification framework that achieves Parameter Space Orthogonality, which is the necessary and sufficient condition for preserving historical performance to the first order. By projecting parameter updates into the JAcobian NUll Space (JANUS), our method significantly recovers compromised historical knowledge without interfering with the underlying fine-tuning process. To overcome the local validity of the Jacobian approximation, we further propose a Multi-step Adaptive Rectification mechanism that utilizes the JANUS shift to dynamically verify the valid trust region and adjust step sizes. Coupled with our proposed ghost projection, ghost orientation comparison, and sequence-level singular value decomposition compression techniques, JANUS also achieves great temporal and spatial efficiency. Experiments demonstrate that JANUS seamlessly integrates with various fine-tuning methods, significantly mitigating the stability-plasticity dilemma by recovering historical knowledge while preserving downstream task adaptation.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.19985 [cs.LG] |
| (or arXiv:2609.19985v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.19985 arXiv-issued DOI via DataCite |
Submission history
From: Zhilong Zheng [view email]
[v1]
Thu, 17 Sep 2026 09:57:46 UTC (482 KB)
[v2]
Wed, 28 Oct 2026 07:23:47 UTC (599 KB)
来源：arXiv cs.AI · arxiv.org