跳到正文
RadarAI· X:UCSD Hao AI Lab (@haoailab)·· 3 小时前AI 评分45

UCSD Hao AI Lab 提出 RecSA:树形稀疏注意力覆盖全上下文

UCSD 提出 RecSA:树形稀疏注意力覆盖全上下文

AI 导读

UCSD Hao AI Lab 提出 RecSA,用树形结构做稀疏注意力,每个节点 16 个子节点,查询每层最多打分 4K 候选并选 256 个向下,上下文每扩大 16 倍树才多一层,64K 以内 2 层、2M 时 4 层。

正文 · 原文

Hao AI Lab· @haoailab · X·2026-11-19 03:20· 1 小时前AI 评分41

AI 导读

UCSD Hao AI Lab 提出 RecSA,用树形结构做稀疏注意力:每个节点 16 个子节点,查询从顶层开始,每层最多打分 4K 候选并选 256 个向下,上下文每扩大 16 倍树才多一层,64K 以内 2 层、2M 时 4 层。

正文 · 原文

Sparse attention such as MoBA and DSA still scores the whole context for every query, so selection gets slower as the context grows. In RecSA, each node of the tree has 16 children. A query starts at the top level, scores at most 4K candidates per level and selects 256 of them to descend into, so the tree grows by one level each time the context grows 16×: 2 levels up to 64K tokens, 4 at 2M. Nodes the query does not select are not dropped. They enter the same softmax as summaries, next to the exact tokens, so every query covers the whole context and needs no sink or local window. At 2M, a query reads about 12K entries, roughly twice as many as at 32K.

来源:Hao AI Lab · x.com

来源:RadarAI · aihot.news