arXiv cs.CL· Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, Ying Wei·· 10 小时前AI 评分38
STEPQuant:Delta 规则循环状态量化中误差何时何地更关键
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
AI 导读
STEPQuant 是面向 Delta 规则循环状态的空间-时间训练后量化框架,按误差幅度与内存生命周期分配精度,并联合拟合 key 行与 value 列缩放。
正文 · 原文
Abstract:Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at this https URL.
| Comments: | Technical report; v2: Expanded experiments and evaluations |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.38169 [cs.CL] |
| (or arXiv:2609.38169v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38169 arXiv-issued DOI via DataCite |
Submission history
From: Haokun Lin [view email]
[v1]
Tue, 29 Sep 2026 17:59:40 UTC (4,628 KB)
[v2]
Thu, 19 Nov 2026 17:27:00 UTC (4,861 KB)
来源:arXiv cs.CL · arxiv.org