arXiv cs.AI· Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su, Han Liu·· 3 天前AI 评分22
基于现代 Hopfield 模型重新审视 Transformer 注意力的统计力学
On the Statistical Mechanics of Attention
AI 导读
研究者通过现代 Hopfield 模型重新审视 Transformer 注意力的统计力学，提出自由能形式化框架来刻画稳健注意力推理的参数区间与临界阈值。该框架将注意力视为统计力学系统的运动方程，并识别出注意力在玻璃态与稳健推理相之间的第一相变，进一步扩展到具有随机 query 和 key 的注意力机制。
正文 · 原文
Abstract:We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
| Subjects: | Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Statistical Mechanics (cond-mat.stat-mech); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.17347 [cs.LG] |
| (or arXiv:2610.17347v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17347 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jerry Yao-Chieh Hu [view email]
[v1]
Mon, 12 Oct 2026 19:55:12 UTC (355 KB)
来源：arXiv cs.AI · arxiv.org