arXiv cs.AI· Asher Labovich·· 11 小时前AI 评分34
循环 Transformer 的稳定性与泛化：基于不动点的分析框架
Stability and Generalization in Looped Transformers
AI 导读
研究者提出基于不动点的框架，从局部吸引性、输入依赖性和几何三个维度分析循环 Transformer 架构的稳定性，并刻画不动点迭代何时产生有意义的预测。理论证明无 recall 的循环网络不动点可数且无法实现强输入依赖性，而带外部归一化的 recall 可同时获得局部吸引、输入局部平滑与稳定反向传播。
正文 · 原文
Abstract:Looped transformers promise test-time compute scaling by spending more iterations on harder problems, but it remains unclear which architectural choices let them extrapolate to harder problems at test time rather than memorize training-specific solutions. We introduce a fixed-point based framework for analyzing looped architectures along three axes of stability -- local attractivity, input-dependence, and geometry -- and use it to characterize when fixed-point iteration yields meaningful predictions. Theoretically, we prove that looped networks without recall have countable fixed points and cannot achieve strong input-dependence at any spectral regime, while recall with outer normalization reliably produces a regime in which fixed points are simultaneously locally-attractive, locally smooth in the input, and supported by stable backpropagation. Empirically, we train small looped transformers on chess, sudoku, and prefix-sums and find that downstream performance tracks the framework's predictions across tasks and architectural configurations. We additionally introduce internal recall, a novel recall placement variant allowing an identity residual path, and exploit its contrasting fixed-point behavior to test our framework's predictions in a controlled setting.
| Comments: | 10 main pages, 30 total. Accepted to NeurIPS 2026 Main Track |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.15259 [cs.LG] |
| (or arXiv:2604.15259v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.15259 arXiv-issued DOI via DataCite |
Submission history
From: Asher Labovich [view email]
[v1]
Thu, 16 Apr 2026 17:35:49 UTC (445 KB)
[v2]
Wed, 22 Apr 2026 15:48:32 UTC (445 KB)
[v3]
Wed, 4 Nov 2026 22:02:46 UTC (388 KB)
来源：arXiv cs.AI · arxiv.org