arXiv cs.MA· Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang·· 3 天前AI 评分41
Cortex：将上下文整合进参数化记忆
Cortex: Consolidating Context into Parametric Memory
AI 导读
受生物系统记忆巩固启发，研究者提出 Memory Consolidation（MC）目标并提出 Cortex，通过生成前瞻性查询预测上下文用途，将上下文条件下的输出蒸馏为可合并的低秩或结构化前馈网络更新。
正文 · 原文
Abstract:Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
| Comments: | Technical report |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2610.20399 [cs.CL] |
| (or arXiv:2610.20399v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20399 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ningyu Zhang [view email]
[v1]
Wed, 14 Oct 2026 16:03:01 UTC (5,250 KB)
来源：arXiv cs.MA · arxiv.org