arXiv cs.CL· Miguel P. Bento, Jo\~ao F. Seabra·· 12 小时前AI 评分26
GaugeQuant：从 LLM 对称性中在线学习量化最优基
GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
AI 导读
GaugeQuant 通过在训练损失中加入 LogSumExp 项打破 Transformer 内部连续对称性，在线选出使激活离群值最小的旋转基，且不改变语言建模目标。该方法无需特定校准数据和量化模拟，仅增加少量推理开销；在 Qwen-2.5 0.5B 上 W4A16 困惑度从 25.62 降至 16.76 并提升零样本准确率，在 LLaMA-2 7B 上困惑度改善但下游准确率未提升。代码已开源。
正文 · 原文
Abstract:Transformers are known to have internal continuous symmetries that leave outputs invariant while altering how activations are distributed across coordinates, and hence how well they quantize. GaugeQuant leverages this in-training by introducing a LogSumExp term to the loss that breaks the symmetries, thus selecting a basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, yielding the language modeling objective completely unaltered. Our quantization strategy requires no specific calibration data, no quantization simulation, and adds a small inference overhead. As a proof of concept, we evaluate on WikiText-2 perplexity and zero-shot reasoning. On Qwen-2.5 0.5B, GaugeQuant reduces perplexity (e.g. $25.62 \to 16.76$ under W4A16) and improves zero-shot accuracy. On LLaMA-2 7B, perplexity improves but downstream accuracy does not. The code is available at: this https URL.
| Comments: | Accepted at the Symmetry and Geometry in Neural Representations Workshop (NeurReps), NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.20757 [cs.LG] |
| (or arXiv:2607.20757v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.20757 arXiv-issued DOI via DataCite |
Submission history
From: Miguel Pedra Bento [view email]
[v1]
Wed, 22 Jul 2026 22:14:23 UTC (86 KB)
[v2]
Fri, 24 Jul 2026 12:57:08 UTC (86 KB)
[v3]
Wed, 4 Nov 2026 15:58:48 UTC (98 KB)
来源：arXiv cs.CL · arxiv.org