跳到正文
arXiv cs.CL· Shauli Ravfogel, Gilad Yehudai, Joan Bruna, Alberto Bietti·· 10 小时前AI 评分34

Transformer 如何以几何方式记忆事实:NeurIPS 2026 论文提出对数嵌入维度即可

Geometric Factual Recall in Transformers

AI 导读

一篇被 NeurIPS 2026 接收的论文提出 Transformer 记忆事实关联的“几何”机制:在单层 Transformer 记忆随机双射的受控设定下,证明对数级嵌入维度即可足够,主体嵌入编码其关联属性向量的线性叠加,小型 MLP 充当按关系条件选择属性的选择器。

正文 · 原文

View PDF HTML (experimental)

Abstract:How do transformer language models memorize factual associations? A common account views internal weight matrices as an associative memory. That account requires parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, \emph{geometric} form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer memorizes random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension is sufficient. Subject embeddings encode \emph{linear superpositions} of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute from the embedding. We extend these results to the multi-hop setting---chains of relational queries such as ``Who is the mother of the wife of $x$?''---and provide constructions with and without chain-of-thought that exhibit a provable capacity--depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with the predicted structure. Causal experiments support the proposed mechanism: when subject embeddings are appropriately re-initialized, the frozen MLP transfers zero-shot to entirely new bijections, showing it has learned a generic selection mechanism rather than functioning as a key-value memory.
Comments: Accepted in Neurips 2026
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2605.12426 [cs.CL]
  (or arXiv:2605.12426v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2605.12426

arXiv-issued DOI via DataCite

Submission history

From: Shauli Ravfogel [view email]
[v1] Tue, 12 May 2026 17:22:22 UTC (2,067 KB)
[v2] Thu, 19 Nov 2026 00:22:50 UTC (2,072 KB)

来源:arXiv cs.CL · arxiv.org