arXiv cs.AI· Afshin Khadangi·· 15 小时前AI 评分38
Continuum Attention：用连续分段线性记忆轨迹替代离散注意力
Continuum Attention
AI 导读
研究者提出 Continuum Attention（CA），让相邻 key 和 value 向量构成连续的分段线性记忆轨迹，每个 query 通过对可见前缀的归一化积分检索信息，段积分以指数矩闭式求解，无需数值积分即可实现解析可微的注意力。
正文 · 原文
Abstract:Conventional causal attention aggregates values at discrete memory positions. We introduce Continuum Attention (CA), in which neighboring key and value vectors define a continuous, piecewise linear memory trajectory and each query retrieves information by normalized integration over its visible prefix. The segment integrals have closed forms in exponential moments, yielding analytically differentiable attention without numerical quadrature. The resulting operator preserves causal masking, normalized nonnegative effective weights, and the standard Transformer decoder structure. Controlled byte level experiments favor exact continuous integration over matched softmax attention. We additionally evaluate data efficiency, out of domain language modeling, zero-shot multiple choice prediction, robustness, and multilingual generalization. In multilingual evaluation, CA consistently outperforms all baselines across every evaluated language, achieving an average 5.8% reduction in bits per byte (BPB) relative to the strongest baseline.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20586 [cs.LG] |
| (or arXiv:2610.20586v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20586 arXiv-issued DOI via DataCite |
Submission history
From: Afshin Khadangi [view email]
[v1]
Wed, 14 Oct 2026 17:57:40 UTC (694 KB)
[v2]
Mon, 26 Oct 2026 15:22:53 UTC (1,731 KB)
来源：arXiv cs.AI · arxiv.org