arXiv cs.CL· Jo\~ao Eduardo Batista, Emil Vatai, Mohamed Wahib·· 11 小时前AI 评分38
无需重训练：大规模混合 LLM 长上下文解码加速方法
Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining
AI 导读
研究者提出一种无需训练或修改权重的组合稀疏注意力方法，加速混合 LLM 中剩余全注意力层的长上下文解码。该方法让每层只关注近期 token 滑动窗口、固定前缀和固定预算的最相关 token，窗口大小与预算通过多目标搜索逐层确定。
正文 · 原文
Abstract:The full-attention layers of large language models (LLMs) make every generated token more expensive than the previous one, slowing models down on long-context tasks. Hybrid models reduce this cost by replacing most full-attention layers with linear-attention layers, but the first still dominate decoding at long contexts. We accelerate these remaining layers without any training or weight changes using combined sparse attention: each layer attends to a sliding window of recent tokens, a pinned prefix, and a fixed budget of most relevant tokens in the context. The window size and budget are found on a per-layer basis using a multi-objective search algorithm. Across six hybrid models from three families, from 4B to 397B parameters, the selected configurations preserve their baseline model's accuracy on 21 long-context datasets. Decoding 64k/256k tokens becomes up to 1.26x/2.22x faster on Qwen3.5-4B, 1.20x/2.23x on Qwen3.5-122B-A10B, and 1.16x/2.06x on Ornith-1.0-397B.
| Comments: | 19 pages (main body: 10 pages, references: 3 pages, appendix: 6 pages), 8 figures, 6 tables |
| Subjects: | Computation and Language (cs.CL); Performance (cs.PF) |
| Cite as: | arXiv:2611.06133 [cs.CL] |
| (or arXiv:2611.06133v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06133 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: João Eduardo Batista [view email]
[v1]
Wed, 4 Nov 2026 08:18:38 UTC (266 KB)
来源：arXiv cs.CL · arxiv.org