arXiv cs.CL· Feichen Song, Boyi Zeng, He Li, Shixiang Song, Yiqin Hao, Zitong Wang, Yizhou Yang, Yifan Liu, Ziwei He, Zhouhan Lin·· 14 小时前AI 评分26
StateLoop：为循环 Transformer 引入持久状态
StateLoop: Persistent State for Looped Transformers
AI 导读
StateLoop 为循环 Transformer 的每个 token 额外维护一份持久状态，通过可学习门控在每次迭代中控制旧状态保留比例与新信息写入量，更新后的状态投影回主干输出用于后续迭代和最终预测。该设计降低了 Pile 困惑度，并在八项零样本任务上提升了平均准确率；相比 Vanilla Looped，其困惑度优势随训练循环深度增加而扩大。
正文 · 原文
Abstract:Looped Transformers offer a parameter-efficient approach to language modeling by repeatedly applying a shared backbone. Vanilla looped Transformers use each backbone output directly as the input to the next iteration. These outputs may contain information useful several iterations later, but retaining it across the loop depends entirely on the single hidden state. We introduce StateLoop, which maintains one additional persistent state per token to retain information across iterations. At each iteration, a learned gate controls how much of the previous state is retained and how much information from the backbone output is incorporated. The updated state is projected back and added to the backbone output for subsequent iterations and final prediction. This design reduces Pile perplexity and improves average accuracy across eight zero-shot tasks over Vanilla Looped. Its perplexity advantage over Vanilla Looped grows with trained loop depth, consistent with persistent state becoming increasingly useful as recurrent computation deepens.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35544 [cs.CL] |
| (or arXiv:2610.35544v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35544 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Feichen Song [view email]
[v1]
Mon, 26 Oct 2026 14:53:49 UTC (1,254 KB)
来源：arXiv cs.CL · arxiv.org