跳到正文
arXiv cs.AI· Aritra Das, Dhruman Gupta, Debayan Gupta·· 10 小时前AI 评分27

线性化 2-Simplicial Attention:用随机特征把三元注意力降到二次复杂度

Linearized 2-Simplicial Attention

AI 导读

研究者提出 2-simplicial attention 的线性化形式,将三线性打分改写为复合 query 与 key 的内积,并用正随机特征近似 key 求和、以固定大小状态存储历史 key,使序列长度上的开销从三次降为二次,且两个 key 都覆盖全部历史 token。

正文 · 原文

View PDF HTML (experimental)

Abstract:We present a linearized form of 2-simplicial attention by rewriting the trilinear score as an inner product between a composite query and a key, so that the sum over keys takes the same form as ordinary softmax attention. We then approximate this sum with positive random features and store all previous keys in a fixed-size state, while the sum over second keys stays exact over all previous tokens. This reduces the cost from cubic to quadratic in sequence length, and both keys range over all previous tokens, which windowed 2-simplicial attention does not allow. We implement it with custom Triton kernels and combine it with Kimi Delta Attention to build a model with no softmax attention at all. With 15B equal training tokens on FineWeb-Edu, this model lowers validation loss from 2.35 to 2.32 over a KDA hybrid and lowers LAMBADA perplexity from 40 to 38.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2608.09307 [cs.AI]
  (or arXiv:2608.09307v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2608.09307

arXiv-issued DOI via DataCite

Submission history

From: Aritra Das [view email]
[v1] Mon, 10 Aug 2026 08:53:30 UTC (20 KB)
[v2] Wed, 18 Nov 2026 16:36:07 UTC (16 KB)

来源:arXiv cs.AI · arxiv.org