arXiv cs.CL· Liqun Ma, Shengkun Tang, Liangyu Wang, Lin Zhao, Wei Luo, Xinjun Niu, Spandan Tiwari, Ashish Sirasao, Kun Zhang, Zhiqiang Shen·· 11 小时前AI 评分31
ResQAdam：面向内存高效量化感知训练的残差保留自适应优化器
ResQAdam: Residual-Preserving Adaptive Optimization for Memory-Efficient Quantization Aware Training
AI 导读
ResQAdam 是一种面向量化感知训练（QAT）的内存高效优化器，相比 AdamW 将优化器状态内存降低 71% 至 74%，训练吞吐下降不到 2%。它只保留单个 BF16 动量，仅在主子空间的 r×r 坐标上应用 Adam，并用残差计算出的归一化符号更新补回被投影方法丢弃的动量信息。
正文 · 原文
Abstract:Quantization-aware training (QAT) is the most reliable way to obtain accurate low-bit large language models (LLMs), but training with AdamW keeps two dense FP32 states for every weight matrix. We find that low-rank memory-efficient optimizers such as GaLore and LDAdam degrade substantially in QAT. A considerable part of the QAT momentum lies outside its top-r singular subspace, and projection-based methods discard this part at every step. We further observe that this residual remains useful when only its sign is kept. Based on these observations, we propose ResQAdam. It keeps a single BF16 momentum, applies Adam only to the r x r coordinates of the principal subspace, and adds a normalized sign update computed from the residual. We evaluate ResQAdam with two from-scratch QAT methods and three QAT fine-tuning pipelines, with weights quantized down to 2 bits. It reduces optimizer-state memory by 71% to 74% relative to AdamW, stays close to AdamW in most settings, and lowers training throughput by less than 2%.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.08924 [cs.LG] |
| (or arXiv:2611.08924v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08924 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shengkun Tang [view email]
[v1]
Fri, 6 Nov 2026 03:12:45 UTC (375 KB)
来源：arXiv cs.CL · arxiv.org