arXiv cs.AI· Yuhan Chen, Siyuan Zhang, Nan Wang, Feiyang Kang, Ruoxi Jia·· 10 天前AI 评分24
Hybrid Latent Attention：面向循环语言模型的高效注意力机制
Hybrid Latent Attention for Looped Language Models
AI 导读
针对循环语言模型 KV cache 随循环次数 T 倍增的问题，研究者提出 Hybrid Latent Attention（HLA），在滑动窗口 W 内保留精确 key/value，窗口外旧 token 压缩为 latent 供各次循环的 query 直接读取。
正文 · 原文
Abstract:Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid Latent Attention (HLA), which keeps exact keys and values within a sliding window of W recent tokens and stores each older token as a compact latent that the query of each loop reads directly, without reconstructing keys and values. We uptrain HLA on Ouro looped models (T=4) with 1.4B and 2.6B parameters, keeping the pretrained weights frozen and training only the added parameters to reproduce the original attention. The cache shrinks by 10.7x per token, fitting 4.0-8.8x as many concurrent sequences per GPU, and decoding throughput improves by 2.5x at 1K-token contexts and by up to 7.4x at 16K. HLA retains over 97% of the original accuracy on math, knowledge and reasoning benchmarks, and 96-100% on long-context retrieval up to 16K tokens. After supervised fine-tuning, it performs on par with the fine-tuned original model on competition-level math.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07940 [cs.CL] |
| (or arXiv:2610.07940v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07940 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhan Chen [view email]
[v1]
Tue, 6 Oct 2026 08:14:07 UTC (185 KB)
来源：arXiv cs.AI · arxiv.org