arXiv cs.CL· Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan, Michael Helcig, Eldar Kurtic, Maximilian Kleinegger, Dan Alistarh·· 10 小时前AI 评分38
GSQ：基于 Gumbel-Softmax 采样的 LLM 高精度低比特标量量化
GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
AI 导读
GSQ 是一种训练后标量量化方法，通过 Gumbel-Softmax 松弛联合学习逐坐标网格分配与逐组缩放，在 Llama-3.1-8B/70B-Instruct 上以 2 比特和 3 比特缩小了标量量化与 QTIP 前沿的差距，并保持与现有标量推理内核兼容。
正文 · 原文
Abstract:Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and "second-generation" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized $\textit{scalar}$ quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus remains compatible with existing scalar inference kernels. We further show that the same discrete-assignment optimization can be applied to practical GGUF K-Quant checkpoints: starting from publicly released GGUF models, GSQ improves accuracy while projecting the result back into the same deployment format. Finally, GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply. The source code is publicly available at this https URL.
| Comments: | Accepted at NeurIPS 2026. 27 pages. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.18556 [cs.CL] |
| (or arXiv:2604.18556v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.18556 arXiv-issued DOI via DataCite |
Submission history
From: Soroush Tabesh [view email]
[v1]
Mon, 20 Apr 2026 17:45:47 UTC (38 KB)
[v2]
Fri, 15 May 2026 09:34:30 UTC (49 KB)
[v3]
Tue, 10 Nov 2026 13:28:54 UTC (52 KB)
来源：arXiv cs.CL · arxiv.org