arXiv cs.CL· Awni Altabaa, Tobias Schnabel, Kiran Tomlinson·· 12 小时前AI 评分31
把 Transformer 理论放到实验里检验：七种理论都无法单独解释模型行为
Putting Transformer Theories to the Test
AI 导读
研究将 Star-free、AC⁰、TC⁰、C-RASP、MPC、BAPO、Sensitivity 七种 Transformer 能力理论映射到统一的任务预测矩阵，并用分布内样本复杂度、长度泛化和预训练语言模型表现做受控实验验证。
正文 · 原文
Abstract:Several formal theories have been developed to characterize the capabilities and limitations of the transformer architecture, each employing different abstractions that capture different features of the architecture. Due to the differences in their underlying assumptions and conceptual frameworks, these theories can disagree in their predictions about what tasks transformers find easy or hard. We map out seven theories of transformer capability (Star-free, $\mathsf{AC}^0$, $\mathsf{TC}^0$, C-RASP, MPC, BAPO, and Sensitivity) and translate their results into a common matrix of predictions across a suite of tasks covering different computational structures. We provide new theoretical results to complete this matrix. We then evaluate theoretical predictions through controlled experiments measuring in-distribution sample complexity, length generalization, and pretrained language model performance. We find that no single theory explains the observed pattern of successes and failures. When a theory does not capture the transformer behaviors relevant to a task, its predictions break down; for instance, we show that the $k$-hop induction task is predicted hard by C-RASP because it does not model content-dependent attention, but we train small transformers on this task that length-generalize. Similarly, theories assuming hard attention like $\mathsf{AC}^0$ and BAPO predict binary majority to be hard, for which we also find success in practice. Together, our results point to several specific features that a theory of transformers must capture: soft rather than hard attention, dynamic content-dependent retrieval, bounded computation at each position, an account of learnability and inductive biases, and a combination of upper and lower bounds.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.32104 [cs.LG] |
| (or arXiv:2610.32104v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32104 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Awni Altabaa [view email]
[v1]
Fri, 23 Oct 2026 03:07:37 UTC (548 KB)
来源：arXiv cs.CL · arxiv.org