arXiv cs.AI· Zeman Li, Ali Behrouz, Yuan Deng, Praneeth Kacham, Rami Al-Rfou, Meisam Razaviyayn, Vahab Mirrokni·· 10 小时前AI 评分42
Toaster:用级联线性记忆与目标传播构建非线性记忆
Toaster: Nonlinear Memory from Cascaded Linear Memories and Target Propagation
AI 导读
Toaster 通过将两个线性记忆级联并在中间加入激活函数,构建出两层非线性记忆,利用目标传播把输出误差投影回第二层记忆来为隐藏层赋值,使每层都能运行未修改的 delta-rule 内核。
正文 · 原文
Abstract:Recurrent memories let language models process long sequences with a fixed-size state. Linear memories such as DeltaNet and KDA store one matrix and take one delta-rule step per token. The product of a chunk of such steps has an exact WY form, so they train fast. Deep memories such as TTT-MLP and Titans store a small MLP, which can hold maps that no matrix can, but their update has no WY form. Toaster bridges the two by building on a known fact: a gradient step on a linear layer is a delta-rule write. It builds a two-layer nonlinear memory from two linear memories with an activation between them. We use target propagation to give the hidden memory its values by projecting the output error back through the second memory. Given a chunk's cross-layer signals, approximated once per chunk, each layer can then run on an unmodified delta-rule kernel. Our default runs this kernel on the output layer and updates the hidden layer once per chunk. Running it on both layers gives a better model in our ablation. At matched backbones and training-token budgets, Toaster has lower loss than KDA at all five tested sizes (0.2B-1.6B non-embedding parameters) on both C4 and PG19. Its C4 perplexity also matches or slightly improves on the attention baseline at every size. At 1.2B it is also below Titans. In a 3:1 hybrid with global attention, the Toaster hybrid has the lowest perplexity of the four hybrids tested, and it is 0.18 (C4) and 0.44 (PG19) below the attention baseline. At the largest size, Toaster's mean three-shot accuracy on 13 tasks is 51.84%, compared with 50.52% for KDA and 50.04% for Titans. In length extrapolation, Toaster's loss stays below KDA's up to 20K tokens (5x the training length). On the 145M backbone, at a hidden width matched to Titans' memory, Toaster takes at most 1.33x KDA's training-step time and is 2.8-5.5x faster than the tested Titans configurations at 32K context.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23648 [cs.LG] |
| (or arXiv:2611.23648v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23648 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zeman Li [view email]
[v1]
Wed, 18 Nov 2026 00:14:27 UTC (233 KB)
来源:arXiv cs.AI · arxiv.org