arXiv cs.AI· Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber·· 3 天前AI 评分54
NoteBank：为线性注意力模型解锁长上下文能力
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
AI 导读
论文提出 NoteBank，一种免训练、即插即用的稀疏情景记忆机制，在未知未来查询前先归档选定文本，用骨干网络原生特征进行写入与寻址，并通过 replay 走原生前向计算读取检索到的证据，无需外部嵌入模型、单独训练的检索器或独立神经阅读器。
正文 · 原文
Authors:Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Piękos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, Jürgen Schmidhuber
Abstract:Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: \emph{recurrent computation need not bear the full burden of long-term storage}. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This \emph{training-free, plug-and-play} interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds $\leqslant$0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.
| Comments: | See our project in this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.15715 [cs.AI] |
| (or arXiv:2610.15715v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15715 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nanbo Li PhD [view email]
[v1]
Sun, 11 Oct 2026 15:58:04 UTC (1,878 KB)
来源：arXiv cs.AI · arxiv.org