跳到正文
arXiv cs.CL· Xinrui Zhong, Fanjiang Ye, Yuke Wang·· 9 小时前AI 评分38

RISA:面向块扩散语言模型的可恢复稀疏注意力

RISA: Recoverable Sparse Attention for Block-Diffusion Language Models

AI 导读

RISA 是一种面向块扩散语言模型的可恢复稀疏注意力方法,通过块内共享读取计划避免重复构建稀疏支持。在 DreamReasoner-8B 上,RISA 保持任务性能相当,32K 上下文下注意力较稠密实现提速 2.67 倍、固定轨迹请求提速 1.36 倍,128K 时注意力提速达 5.53 倍。

正文 · 原文

View PDF HTML (experimental)

Abstract:Block-wise diffusion language models (DLMs) refine multiple tokens in parallel, offering an alternative to token-by-token autoregressive decoding. Their long-context inference, however, remains bottlenecked by memory-bound attention: every denoising step repeatedly reads a growing historical key--value (KV) cache. Directly applying sparse attention does not fully solve this problem, because different queries can select different pages, inflating the block's collective read set, while repeated support construction adds overhead and permanent eviction can discard information needed later. We present RISA, a recoverable sparse-attention method with shared read plans for block-wise DLMs. RISA exploits the fact that historical KV remains fixed within a block while queries evolve across denoising steps. Newly generated history is organized into exact priority and recovery tiers; seed queries then build a protected, bounded read plan whose indices are reused across the block while attention is recomputed with fresh queries and current-block KV. This separates history preservation from repeated reading and amortizes sparse-support construction without changing the backbone or decoder. On DreamReasoner-8B, RISA retains comparable task performance. At 32K, the default configuration speeds up attention by $2.67\times$ and fixed-trajectory requests by $1.36\times$ over dense, and attention speedup reaches $5.53\times$ at 128K. Across eight settings on three models, RISA decodes faster than the tested MAGE, QUEST, and LoSA budgets at matched KL.
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2611.22191 [cs.CL]
  (or arXiv:2611.22191v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.22191

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xinrui Zhong [view email]
[v1] Tue, 17 Nov 2026 04:27:48 UTC (314 KB)

来源:arXiv cs.CL · arxiv.org