arXiv cs.CL· Xu Yang, Jiapeng Zhang, Yuxin Chen, Feiqiang Sun, Chengguang Xu, Feng Jin, Zhuo Tang·· 10 天前AI 评分29
Self-Indexing Attention：压缩兼容的稀疏长上下文 LLM 推理
Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference
AI 导读
研究者提出 Self-Indexing Attention，一个无需训练的稀疏长上下文推理框架，基于共享的变换域符号-幅值表示，用 1-bit 索引同时支持分组 prefill 选择和 decode 检索。
正文 · 原文
Abstract:Sparse long-context inference requires efficient token retrieval in both prefill and decode. Existing methods often use different retrieval strategies for the two stages, preventing one retrieval representation from being reused throughout inference. We propose Self-Indexing Attention, a training-free framework built on a shared transform-domain sign-magnitude representation. The key signs provide a reusable token-level index for grouped prefill selection and decode retrieval, while the same representation remains compatible with external KV-cache compression without separate indexer metadata. This 1-bit index enables efficient retrieval through bitwise operations widely supported by modern accelerators. At 5% attention density, Self-Indexing Attention remains close to dense attention on LongBench and RULER and achieves up to 6.1x prefill and 10.3x decode attention-operator speedups. Experiments with TurboQuant and DeepSeekV4-Flash further demonstrate compatibility with low-bit KV-cache compression and pretrained sparse-attention indexers.
| Subjects: | Information Retrieval (cs.IR); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.13205 [cs.IR] |
| (or arXiv:2609.13205v2 [cs.IR] for this version) | |
| https://doi.org/10.48550/arXiv.2609.13205 arXiv-issued DOI via DataCite |
Submission history
From: Xu Yang [view email]
[v1]
Sun, 16 Aug 2026 03:41:37 UTC (214 KB)
[v2]
Wed, 7 Oct 2026 07:41:29 UTC (472 KB)
来源：arXiv cs.CL · arxiv.org