arXiv cs.CL· Melody Zixuan Li, Taylor Webb·· 10 小时前AI 评分44
是什么驱动了大语言模型中符号机制的出现？
What drives the emergence of symbolic mechanisms in large language models?
AI 导读
研究通过从头训练神经网络发现，数据多样性而非架构是神经网络涌现符号计算的主要驱动因素。在多种数据多样性维度上，多样性越高符号特征越强；相同特征甚至出现在修改后的 Transformer 架构和多层感知机中，说明架构归纳偏置并非必要条件。该结论在实体-属性绑定任务中同样成立，论文将发表于 NeurIPS 2026。
正文 · 原文
Abstract:Recent work has identified a set of emergent symbolic mechanisms that support abstract reasoning in large language models, but it remains unclear what factors drive the emergence of these mechanisms. To address this question, we trained neural networks from scratch on abstract sequence tasks, varying both architectural and data distributional factors, and investigated the effects of these factors on the emergence of symbolic mechanisms. Using representational, attentional, and causal mediation analyses, we first confirmed that transformer-based language models trained from scratch on our task developed symbolic mechanisms. We then found that, across multiple axes of data diversity, greater diversity produced stronger symbolic signatures. Surprisingly, the same signatures emerged in modified transformer architectures and even multilayer perceptrons, indicating that architectural inductive biases are not necessary for the development of symbolic mechanisms. We also observed consistent patterns in an entity-attribute binding task. Together, these findings identify data diversity, not architecture, as the primary driver of emergent symbolic computation in neural networks.
| Comments: | 25 pages, 15 figures, 5 appendices. To appear at NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.13270 [cs.CL] |
| (or arXiv:2611.13270v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13270 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zixuan Li [view email]
[v1]
Mon, 9 Nov 2026 21:36:57 UTC (1,373 KB)
来源：arXiv cs.CL · arxiv.org