arXiv cs.AI· Tianqi Chen, Xunchao Li, Linchuan Xie, Guanghua Yu, Jianchen Zhu, Peisong Wang, Jian Cheng·· 13 小时前AI 评分32
GateRQ：按门控决定量化条件记忆的每次查找精度
Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs
AI 导读
GateRQ 利用条件记忆层中缩放检索行的门控值来分配量化精度，在 DeepSeek-V4.1-Flash 上无需重训练即将 GPU 表内存降低 7.8 倍，vLLM 吞吐量较 FP8 表提升 11%-31%。
正文 · 原文
Abstract:Conditional memory layers, such as DeepSeek's Engram, attach large hashed n-gram tables to a language model, and removing them from DeepSeek-V4.1-Flash costs 36.8 PopQA points. Low-bit quantization loses long-tail facts first, and the standard remedy, giving more bits to frequently accessed rows, backfires: refining lookups by access frequency is worse than refining them at random. We trace this to the gate that scales each retrieved row. A lookup's quantization error is weighted by its gate, which is small for frequent n-grams, large for the rare n-grams of long-tail entities, and computable from a 1-bit code. GateRQ therefore stores each row as a ladder of 1-bit residual codes, keeps level 1 in GPU memory to compute the gate, and reads deeper levels only for the lookups the gate ranks highest. Without retraining, GateRQ cuts GPU table memory by 7.8x on DeepSeek-V4.1-Flash, which raises vLLM throughput by 11%-31% over FP8 tables, while losing about 1 PopQA point at 1.50 bits read per lookup and 0.2 points at 2.10 bits across two models.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35985 [cs.LG] |
| (or arXiv:2610.35985v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35985 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chen Tianqi [view email]
[v1]
Mon, 26 Oct 2026 11:53:58 UTC (401 KB)
来源：arXiv cs.AI · arxiv.org