arXiv cs.CL· Lele Yang, Muxi Diao, Zhonghao Yan, Jiankang Chen, Bin Wen, Tingting Gao, Wentao Zhang, Kongming Liang, Zhanyu Ma, Weiran Xu·· 12 小时前AI 评分29
LooM：面向因果编码器-解码器语言模型的循环不变记忆机制
LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models
AI 导读
LooM 提出一种循环不变记忆机制，在因果编码器-解码器中仅循环第一层之后的解码器层，每层只在首次访问某个 token 时写入 KV 条目，使缓存不随循环次数增长。该设计支持每个 token 3 至 21 次层执行的任意预算，单个在随机预算上训练的模型即可覆盖全部预算且无需路由器。在相同缓存大小下，LooM 在语言建模、零样本准确率和长上下文困惑度上均优于单次前向基线。
正文 · 原文
Abstract:Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training. We propose LooM (Loop-invariant Memory), which addresses each in turn. In a causal encoder-decoder (CED), LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value (KV) entry for a token only on its first visit, so the cache does not grow with the loop count. These layers loop independently, providing every budget from 3 to 21 layer executions per token, and one model trained on randomly sampled budgets covers all of them without a router. With enough loops, LooM outperforms a single-pass baseline trained on the same data on language modeling, zero-shot accuracy and long-context perplexity, while using the same cache size. Our analysis further shows that the training budget distribution decides what extra depth buys. These results suggest that a CED is a natural place to loop: memory is written once, and depth becomes a budget chosen at deployment.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.06835 [cs.CL] |
| (or arXiv:2611.06835v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06835 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Muxi Diao [view email]
[v1]
Wed, 4 Nov 2026 17:52:11 UTC (120 KB)
来源：arXiv cs.CL · arxiv.org