arXiv cs.AI· Bruce Changlong Xu·· 12 小时前AI 评分31
激活敏感度作为训练后量化统一原理：AWQ 与 GPTQ 被证明是同一损失二次型的特例
Activation Sensitivity as a Unifying Principle for Post-Training Quantization
AI 导读
研究证明 AWQ、GPTQ、对角 Fisher 量化、块重建与 Hadamard 旋转等训练后量化方法，都是梯度加权激活敏感度所支配的同一损失二次型的特例。基于该框架提出的 SAQ-GPTQ 在 token 梯度异质条件下保持 >99% 关键通道召回率，INT3/INT4 额外损失比未加权 GPTQ 降低约 3 倍。
正文 · 原文
Abstract:Post-training quantization (PTQ) methods for large language models compress pretrained weights to low bit-widths without retraining by optimizing layer-local proxy objectives. Activation-aware scaling methods such as AWQ protect input channels with large activation norms, whereas second-order error-compensation methods such as GPTQ redistribute rounding error using the inverse of the unweighted input covariance matrix. We show that both paradigms -- along with diagonal Fisher quantization, block reconstruction, and Hadamard rotation methods -- are special cases of minimizing a single loss-derived quadratic form governed by gradient-weighted activation sensitivity. From a Gauss-Newton expansion of the task loss at a linear layer $Y = X W^\top$, we prove that under isotropic output curvature the expected excess loss from weight perturbation $\Delta W$ reduces to $\frac{1}{2 d_{\mathrm{out}}}\operatorname{Tr}(\Delta W H_D \Delta W^\top)$, where $H_D = \frac{1}{n} X^\top D X$ weights each calibration token's outer product by its squared output-gradient norm $D_{tt} = \|\nabla_{y_t}\mathcal{L}\|_2^2$. Within this framework, AWQ corresponds to a diagonal approximation of $H_D$ with uniform token weights ($D = I_n$) optimized via channel scaling, while GPTQ corresponds to Cholesky error compensation on $H_D$ under $D = I_n$. We derive closed-form optimal scaling exponents for diagonal sensitivity protection, formulate sensitivity-weighted Cholesky quantization (SAQ-GPTQ) on $H_D$, and prove bounds on token-gradient heterogeneity bias, cross-layer error amplification across residual blocks, and finite-sample calibration concentration. Controlled numerical simulations confirm that under token-gradient heterogeneity, replacing $X^\top X$ with $X^\top D X$ preserves $>99\%$ top-channel saliency recall and reduces INT3/INT4 excess loss by roughly $3\times$ over unweighted GPTQ.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.6; G.1.6 |
| Cite as: | arXiv:2601.11663 [cs.LG] |
| (or arXiv:2601.11663v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.11663 arXiv-issued DOI via DataCite |
Submission history
From: Bruce Xu [view email]
[v1]
Thu, 15 Jan 2026 22:27:19 UTC (16 KB)
[v2]
Tue, 20 Oct 2026 06:57:59 UTC (21 KB)
来源：arXiv cs.AI · arxiv.org