arXiv cs.AI· Shuoran Jiang, Wentao Zhu, Yang Xiang, Dongfeng Li, Yunan Zhang, Mengchen Zhao, Yuhui Guo, Kai Chen, Youcheng Pan, Xiangping Wu, Qingcai Chen·· 2 天前AI 评分26
面向大语言模型高效持续学习的功能子网络定位
Functional Subnetwork Localization for Efficient Continual Learning in Large Language Models
AI 导读
研究者提出 CiMEO 持续学习框架，通过梯度归因定位 LLM 中互重叠极低的功能子网络，将每个训练样本路由到对应子网络，仅更新选定区域的参数与优化器状态。在 140 万条序列指令上，CiMEO 在编程、考试类任务和数学推理五个生成式基准上取得对比方法中最高的平均性能，并缓解灾难性遗忘。子空间重叠与主角度分析表明，局部化更新有助于保留预训练表示。
正文 · 原文
Authors:Shuoran Jiang, Wentao Zhu, Yang Xiang, Dongfeng Li, Yunan Zhang, Mengchen Zhao, Yuhui Guo, Kai Chen, Youcheng Pan, Xiangping Wu, Qingcai Chen
Abstract:Biological intelligence continually adapts to new information while retaining prior skills through localized plasticity. In contrast, continual learning in large language models (LLMs) remains vulnerable to catastrophic forgetting. A promising direction is to identify functionally specialized subnetworks within LLMs that enable targeted and resource-efficient adaptation. Using gradient-attribution profiling, we localize cognition-associated functional subnetworks exhibiting extremely low mutual overlap. Building on this finding, we propose Function-isolated Memory-Efficient Optimization (CiMEO), a continual learning framework that routes each training example to its corresponding functional subnetwork, restricting parameter updates and optimizer states to selected regions. Across 1.4 million sequential instructions, CiMEO achieves the highest average performance among the compared methods on five generative benchmarks spanning programming, examination-style tasks, and mathematical reasoning, while mitigating catastrophic forgetting. Subspace-overlap and principal-angle analyses further indicate that localized updates help preserve pre-trained representations, highlighting functional subnetwork localization as a practical approach to efficient and stable continual learning in LLMs.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.21537 [cs.AI] |
| (or arXiv:2610.21537v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21537 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuoran Jiang [view email]
[v1]
Thu, 15 Oct 2026 09:03:08 UTC (6,954 KB)
来源：arXiv cs.AI · arxiv.org