RAM-Net 如何桥接 KV-Cache 量化与线性注意力：从理论到预训练权重迁移

View PDF HTML (experimental)

Abstract:KV-cache quantization and linear attention are two representative approaches to tackling the storage and computational costs of Transformers. KV-cache quantization compresses individual KV entries into discrete codes but retains all entries, whereas linear attention recurrently aggregates multiple historical KV contributions into a fixed-size continuous state but can introduce interference. This contrast raises the question of whether per-KV compression and multi-KV aggregation can be bridged within a single mechanism for efficient attention. We identify RAM-Net as such a bridge through soft assignments over a discrete address space. These assignments determine recurrent updates to the continuous slot state associated with each address. Under a restricted RAM-Net construction, we prove that soft address assignments extend hard quantized matching to a separable read-write overlap that locally approximates full-attention similarity and supports recurrent aggregation. These connections further enable Transformer-to-RAM-Net weight migration through a new path based on a soft-quantized intermediate construction. Across nine pretrained Transformer models from 0.3B to 7B parameters, RAM-Net recovers an average of 87.1% of the teachers' accuracy gains over random guessing across six commonsense and knowledge tasks using only a 500M-token budget per model.

Subjects:	Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as:	arXiv:2610.11214 [cs.LG]
	(or arXiv:2610.11214v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.11214 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Kaicheng Xiao [view email]
[v1] Thu, 8 Oct 2026 04:12:49 UTC (10,602 KB)