arXiv cs.CL· Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai·· 2 天前AI 评分58
Loopie 开源循环 Transformer：20B-A2B 与 6B-A0.6B 两款 MoE 模型
Loop the Loopies!
AI 导读
论文提出开源循环 Transformer 系列 Loopie，包含 20B 参数、2B 激活参数的 MoE 模型和 6B 参数、0.6B 激活参数的 MoE 模型。
正文 · 原文
Abstract:We present Loopie, the most powerful open-source looped Transformer to date. The Loopie series consists of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N-fold pre-training compute, increasing the parameter count by a factor of N usually outperforms looping N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.16051 [cs.CL] |
| (or arXiv:2607.16051v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.16051 arXiv-issued DOI via DataCite |
Submission history
From: Zitian Gao [view email]
[v1]
Fri, 17 Jul 2026 15:28:43 UTC (829 KB)
[v2]
Mon, 20 Jul 2026 15:59:50 UTC (835 KB)
[v3]
Mon, 26 Oct 2026 17:12:57 UTC (1,055 KB)
来源：arXiv cs.CL · arxiv.org