arXiv cs.CL· Jiedong Zhuang, Peng Zhao, Jialin Tang, Haoji Hu·· 11 小时前AI 评分31
SparseGDN：线性注意力中的分块稀疏机制
SparseGDN: Chunk Sparse in Linear Attention
AI 导读
SparseGDN 是一个硬件感知框架，将结构化块内近似与输入相关的循环事件选择结合，在 Qwen3.5-4B 和 Qwen3.5-9B 上测试 4K 至 128K 上下文，平均保留约 29% 的循环事件，实现 1.50-1.94 倍 GDN-core prefill 加速（含在线选择开销）。
正文 · 原文
Abstract:Gated Delta Networks (GDNs) offer linear-time sequence modeling, yet long-context prefill remains costly due to dense intra-chunk computation, recurrent-state updates, and intermediate state storage. We introduce SparseGDN, a hardware-aware framework that combines structured intra-chunk approximation with input-dependent recurrent-event selection while producing outputs for every token. A block-adjacent solver reduces local Delta-solve computation, while a dynamic selector allocates recurrent-update budgets across head pairs. The GPU implementation materializes updates only for selected events and reconstructs token outputs using compact state snapshots and group-local interactions, reducing computation and memory traffic. We evaluate SparseGDN on Qwen3.5-4B and Qwen3.5-9B across RULER tasks and context lengths from 4K to 128K. Retaining approximately 29% of recurrent events on average, SparseGDN achieves 1.50-1.94x GDN-core prefill speedups over native FLA kernels, including online selection overhead.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.41420 [cs.LG] |
| (or arXiv:2610.41420v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41420 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiedong Zhuang [view email]
[v1]
Fri, 30 Oct 2026 08:27:36 UTC (681 KB)
来源：arXiv cs.CL · arxiv.org