跳到正文
arXiv cs.CL· Zeyuan Allen-Zhu·· 10 小时前AI 评分58

语言模型物理学 Part 4.2:Canon 层在万亿 token 规模下让合成预训练在现实中生效

Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

AI 导读

Zeyuan Allen-Zhu 用 64 次从零预训练实验(1.3-8B 参数、1-2T Nemotron-CC token)研究标准基准能否反映架构差异。

正文 · 原文

View PDF HTML (experimental)

Abstract:Do standard benchmarks see the architectural differences that matter? We study it with 64 from-scratch pretraining runs of Transformers and three linear-model families at 1.3-8B parameters on 1-2T Nemotron-CC tokens, at a cost of roughly 100k USD per 8B run.
In academic-scale pretraining (1.3B parameters, 100B tokens), architectural differences can all but disappear when the models are carefully tuned. Instead, at 1-2T tokens they become measurable: Canon layers, which add local token mixing, consistently improve Transformers, saving about 30% of pretraining tokens for the same MMLU accuracy at 8B. We also introduce GLA5 and GDN2, which improve gated linear attention and Gated DeltaNet with low-rank output gates. Surprisingly, with Canon, the simplest linear model, gated linear attention, matches Gated DeltaNet and outperforms Mamba2 even at 8B/1T. Therefore, the extra machinery of Gated DeltaNet and Mamba2 may do little more than mix neighboring tokens.
Even at 8B/1T, however, high scores on standard benchmarks can hide weaknesses in basic in-context skills. In particular, linear models that match or beat Transformers on these benchmarks cannot, in fact, reliably retrieve information from a short 500-token context. Moreover, no model, Transformer or linear, reliably beats random guessing on arguably the simplest two-hop question, stated in a few short sentences, even at this scale.
These findings $\textbf{challenge traditional architecture research}$ based primarily on standard benchmark scores. In contrast, small synthetic playgrounds, which predicted the architectural differences above at <100M parameters, can reveal what these benchmarks miss. For practitioners, one lesson is to include in-context reasoning examples in pretraining data from an early stage, rather than reserve them for later mid-training, let alone post-training.
Comments: Results, recipes, and code have been public since December 9, 2025 at this https URL
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24963 [cs.CL]
  (or arXiv:2611.24963v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.24963

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zeyuan Allen-Zhu [view email]
[v1] Thu, 12 Nov 2026 18:51:19 UTC (1,006 KB)

来源:arXiv cs.CL · arxiv.org