Adamas：面向高效长上下文推理的 Hadamard 稀疏注意力机制

View PDF HTML (experimental)

Abstract:Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods reduce these costs by attending to only a subset of key-value (KV) tokens, but fixed or coarse-grained selection strategies can miss important tokens or retain substantial redundancy under constrained token budgets. We introduce \textbf{Adamas}, an accurate and efficient token-level sparse attention mechanism for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and uses lightweight distance-based estimation for efficient top-$k$ candidate selection. The selected positions are then used to gather the corresponding original KV states for sparse attention. Comprehensive evaluations show that Adamas maintains near-lossless performance within the effective context length of Llama-3.1-8B-Instruct. In efficiency evaluations, Adamas achieves up to $6.5\times$ self-attention speedup and $1.7\times$ end-to-end decoding speedup at a 32K context length. Adamas also maintains performance comparable to full attention on challenging long-form reasoning benchmarks, including AIME 2024 and AIME 2025.

Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2510.18413 [cs.CL]
	(or arXiv:2510.18413v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2510.18413 arXiv-issued DOI via DataCite

Submission history

From: Siyuan Yan [view email]
[v1] Tue, 21 Oct 2025 08:44:47 UTC (1,170 KB)
[v2] Fri, 6 Nov 2026 10:07:01 UTC (2,681 KB)