arXiv cs.AI· Dan Lee, Seungwook Han, Akarsh Kumar, Pulkit Agrawal·· 1 天前AI 评分48
用神经细胞自动机预预训练语言模型：164M 合成 token 提升下游性能 6%
Training Language Models via Neural Cellular Automata
AI 导读
研究者提出用神经细胞自动机（NCA）生成非语言合成数据，对 LLM 做"预预训练"，即在自然语言预训练之前先训练合成数据。仅用 164M NCA token，在 400M-3B 模型上使下游语言建模性能最高提升 6%，收敛速度最高加快 1.6x，效果甚至超过用 1.6B 高质量自然语言 token 预预训练。
正文 · 原文
Abstract:Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, contains human biases, and entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs--training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x in 400M-3B models trained with Chinchilla optimal data budgets. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of high-quality natural language data with more compute. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.
| Comments: | Website: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.10055 [cs.LG] |
| (or arXiv:2603.10055v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.10055 arXiv-issued DOI via DataCite |
Submission history
From: Seungwook Han [view email]
[v1]
Mon, 9 Mar 2026 18:14:26 UTC (1,417 KB)
[v2]
Wed, 11 Nov 2026 17:22:52 UTC (2,827 KB)
来源：arXiv cs.AI · arxiv.org