跳到正文
arXiv cs.AI· Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Huaqing Zheng, Xindian Ma, Peng Zhang·· 10 小时前AI 评分38

SharQ:打通激活稀疏与 FP4 量化,加速 LLM 推理

SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

AI 导读

SharQ 是一种免训练的 LLM 推理方法,通过在线稀疏—稠密分解把激活稀疏性与 FP4 量化结合,无需校准数据、重训练或模型特定调优。在 Llama-3.1-8B、Qwen2.5-7B、Qwen3-30B-A3B 和 Qwen3-VL-8B 上,它挽回了 NVFP4 与 FP16 之间 43–63% 的精度差距,并兼容 NVFP4、HiF4、MXFP4 格式。

正文 · 原文

View PDF HTML (experimental)

Abstract:Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error. We introduce SharQ, a training-free inference method that bridges activation sparsity and FP4 quantization through an online sparse--dense decomposition. For each activation tensor, SharQ generates an input-adaptive N:M mask to extract an outlier-dominated sparse backbone, quantizes it to FP4, and defines a dense residual relative to the quantized sparse backbone rather than the unquantized sparse values. A sparse FP4 GEMM processes the backbone while a dense FP4 GEMM compensates for both mask-induced activation loss and sparse-path quantization error. The two paths share a single FP4 weight payload with path-specific scale views, and a fused preparation kernel absorbs mask generation, residual construction, and layer normalization into one operator. SharQ requires no calibration data, retraining, or model-specific tuning. Evaluated on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, and Qwen3-VL-8B, SharQ recovers 43--63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, and generalizes across NVFP4, HiF4, and MXFP4 formats. On an RTX 5090, SharQ delivers 2.2--2.4$\times$ latency reduction over FP16 and 1.2--1.4$\times$ throughput improvement over FP8 in language model serving, and up to 1.58$\times$ speedup on Wan2.2-T2V-A14B video generation when combined with SageAttention. Our code is available at this https URL.
Comments: 14 pages, 4 figures
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2606.26587 [cs.LG]
  (or arXiv:2606.26587v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2606.26587

arXiv-issued DOI via DataCite

Submission history

From: Haoqian Meng [view email]
[v1] Thu, 25 Jun 2026 04:19:04 UTC (1,067 KB)
[v2] Wed, 18 Nov 2026 08:51:32 UTC (798 KB)

来源:arXiv cs.AI · arxiv.org