arXiv cs.AI· Haowei Chen, Yichen Wang, Congfu Xu·· 10 小时前AI 评分27
Kernel Ridge Attention(KRA):带选择性遗忘的非线性记忆序列混合器
Kernel Ridge Attention: Nonlinear Memory with Selective Forgetting
AI 导读
Kernel Ridge Attention(KRA)是一种循环序列混合器,通过递归岭回归存储关联,并用内容相关的 Ornstein-Uhlenbeck 转移控制记忆保留与不确定性恢复。其推理状态大小与上下文长度无关,借助关联块映射与显式伴随实现并行训练。在受控语言建模实验中,KRA 的预测质量优于注意力与循环基线,并在所评估的循环模型中增强了自然联想召回。
正文 · 原文
Abstract:Kernel Ridge Attention (KRA) is a recurrent sequence mixer that combines learned nonlinear features with covariance-guided memory updates. It stores associations through recursive ridge regression, using correlations among past keys to determine how new observations modify memory. A content-dependent Ornstein--Uhlenbeck transition controls retention and restores uncertainty for future writes. This construction admits an exact temporal-kernel interpretation and maintains an inference state whose size is independent of context length. Associative block maps and explicit adjoints support parallel training while preserving the recursive update. In controlled language-modeling experiments, KRA improves prediction quality over attention and recurrent baselines, strengthens natural associative recall over the evaluated recurrent models, and achieves practical distributed training throughput. These properties provide a principled foundation for nonlinear, adaptive sequence memory.
| Comments: | 15 pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23896 [cs.LG] |
| (or arXiv:2611.23896v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23896 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yichen Wang [view email]
[v1]
Wed, 18 Nov 2026 05:49:34 UTC (99 KB)
来源:arXiv cs.AI · arxiv.org