arXiv cs.CL· Shauli Ravfogel, Gilad Yehudai, Joan Bruna, Alberto Bietti·· 10 小时前AI 评分34
Transformer 如何以几何方式记忆事实:NeurIPS 2026 论文提出对数嵌入维度即可
Geometric Factual Recall in Transformers
AI 导读
一篇被 NeurIPS 2026 接收的论文提出 Transformer 记忆事实关联的“几何”机制:在单层 Transformer 记忆随机双射的受控设定下,证明对数级嵌入维度即可足够,主体嵌入编码其关联属性向量的线性叠加,小型 MLP 充当按关系条件选择属性的选择器。
正文 · 原文
Abstract:How do transformer language models memorize factual associations? A common account views internal weight matrices as an associative memory. That account requires parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, \emph{geometric} form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer memorizes random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension is sufficient. Subject embeddings encode \emph{linear superpositions} of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute from the embedding. We extend these results to the multi-hop setting---chains of relational queries such as ``Who is the mother of the wife of $x$?''---and provide constructions with and without chain-of-thought that exhibit a provable capacity--depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with the predicted structure. Causal experiments support the proposed mechanism: when subject embeddings are appropriately re-initialized, the frozen MLP transfers zero-shot to entirely new bijections, showing it has learned a generic selection mechanism rather than functioning as a key-value memory.
| Comments: | Accepted in Neurips 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.12426 [cs.CL] |
| (or arXiv:2605.12426v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12426 arXiv-issued DOI via DataCite |
Submission history
From: Shauli Ravfogel [view email]
[v1]
Tue, 12 May 2026 17:22:22 UTC (2,067 KB)
[v2]
Thu, 19 Nov 2026 00:22:50 UTC (2,072 KB)
来源:arXiv cs.CL · arxiv.org