arXiv cs.AI· Alexander Stern, Tim Rockt\"aschel, Acyr Locatelli, Laura Ruis·· 14 小时前AI 评分38
重新审视预预训练：合成数据阶段难超纯自然语言训练
Rethinking Pre-pretraining
AI 导读
计算量匹配的实验显示，将算力投入合成预预训练（PPT）在四种课程与三种训练机制下均未超过纯自然语言训练。在 881M 参数、50B token 的高 token/参数比设置中，自然语言稀缺时重复 100M token 语料反而优于所有合成课程。此前报告的 PPT 收益对权重初始化方案敏感，NCA 复现中原始初始化下可获 5.4% 困惑度改善，且模型会在后续自然语言训练中遗忘 PPT 所学内容。
正文 · 原文
Abstract:We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments. This holds across four PPT curricula (k-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes: data-constrained, data-unconstrained, and a high token-to-parameter-ratio regime (881M parameters trained on 50B tokens). When natural language is scarce, repeated epochs over a 100M-token corpus beat every synthetic curriculum on natural-language validation loss, with no consistent downstream accuracy advantage from PPT. We find that PPT gains reported in prior work are sensitive to the weight initialization scheme. In an NCA reproduction, we recover a 5.4% perplexity improvement from PPT under the original initialization scheme. An alternative scheme improves both methods, but the no-PPT baseline more, closing the gap between them. However, when PPT receives additional compute rather than a share of the natural-language budget, gains persist at small natural-language budgets even with improved initialization, but disappear by 1B tokens. Finally, models forget what they learn during PPT in subsequent natural-language training; interleaving the same data throughout training prevents forgetting, but does not significantly improve natural-language performance. Our results suggest limited benefits of synthetic pre-pretraining as a distinct training stage and motivate rethinking how such data are used in language model training.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.40051 [cs.CL] |
| (or arXiv:2610.40051v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40051 arXiv-issued DOI via DataCite |
Submission history
From: Alexander Stern [view email]
[v1]
Thu, 29 Oct 2026 13:45:58 UTC (745 KB)
来源：arXiv cs.AI · arxiv.org