arXiv cs.CL· Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, Huan Wang·· 9 天前AI 评分27
SparseDecoding：面向解码的剪枝框架，实现精准高效的 LLM 推理
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
AI 导读
SparseDecoding 是一个面向解码的 LLM 剪枝框架，通过用稠密模型自回归生成（排除 prefill）阶段的逐层激活构建校准矩阵，使剪枝目标与解码激活对齐。该方法在 Llama-3.1-8B、Llama-3.3-70B、Qwen3-14B / 32B 上长文本生成基准中持续优于固定文本校准，并在 A100 GPU 上实现最高 1.48x 端到端解码加速。
正文 · 原文
Abstract:The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.12327 [cs.LG] |
| (or arXiv:2610.12327v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12327 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qitong Wang [view email]
[v1]
Thu, 8 Oct 2026 17:06:32 UTC (233 KB)
来源：arXiv cs.CL · arxiv.org