跳到正文
arXiv cs.CL· Albert Alcalde, Giovanni Fantuzzi, Enrique Zuazua·· 9 小时前AI 评分26

纯注意力 hardmax Transformer 中的聚类现象及其在情感分析中的作用

Clustering in pure-attention hardmax transformers and its role in sentiment analysis

AI 导读

研究严格刻画了采用 hardmax 自注意力与归一化子层的 Transformer 在层数趋于无穷时的行为,证明输入会渐近收敛到由“leader”点决定的聚类平衡态。基于该理论,作者构建了一个完全可解释的 Transformer 模型用于情感分析,通过将无意义词围绕承载语义的 leader 词聚类来捕捉上下文。

正文 · 原文

View PDF HTML (experimental)

Abstract:Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called \textit{leaders}. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.
Comments: Funded by the European Union's HORIZON-MSCA-2021-DN-01-01, through the ModConFlex project ("Modelling and control of flexible structures interacting with fluids''; Grant Agreement No. 101073558; DOI: this https URL). Accompanying code available at: this https URL, code DOI: this https URL
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG); Dynamical Systems (math.DS); Machine Learning (stat.ML)
MSC classes: 68T07, 68T50
Cite as: arXiv:2407.01602 [cs.CL]
  (or arXiv:2407.01602v3 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2407.01602

arXiv-issued DOI via DataCite

Journal reference: SIAM Journal on Mathematics of Data Science 7(3):1367-1393, 2025
Related DOI: https://doi.org/10.1137/24M167086X

DOI(s) linking to related resources

Submission history

From: Albert Alcalde [view email]
[v1] Wed, 26 Jun 2024 16:13:35 UTC (394 KB)
[v2] Wed, 13 May 2026 09:54:58 UTC (749 KB)
[v3] Tue, 17 Nov 2026 15:11:58 UTC (749 KB)

来源:arXiv cs.CL · arxiv.org