arXiv cs.AI· Michela Lapenna, Rita Fioresi, Bahman Gharesifard·· 10 天前AI 评分22
Sinkhorn 双随机注意力秩衰减分析
Sinkhorn doubly stochastic attention rank decay analysis
AI 导读
研究发现，用 Sinkhorn 算法归一化的双随机注意力矩阵比标准 softmax 行随机注意力更能有效保持秩，缓解 Transformer 随深度加深的秩坍缩问题。作者在情感分析和图像分类任务上验证了这一现象，并推导出纯自注意力下 Sinkhorn 归一化的秩衰减理论上界：秩随深度呈双指数衰减至 1，与 softmax 已有结论一致。
正文 · 原文
Abstract:The self-attention mechanism is central to the success of Transformer architectures. However, standard row-stochastic attention has been shown to suffer from significant signal degradation across layers. In particular, it can induce rank collapse, resulting in increasingly uniform token representations, as well as entropy collapse, characterized by highly concentrated attention distributions. Recent work has highlighted the benefits of doubly stochastic attention as a form of entropy regularization, promoting a more balanced attention distribution and leading to improved empirical performance. In this paper, we study rank collapse across network depth and show that doubly stochastic attention matrices normalized with Sinkhorn algorithm preserve rank more effectively than standard softmax row-stochastic ones. As previously shown for softmax, skip connections are crucial to mitigate rank collapse. We empirically validate this phenomenon on both sentiment analysis and image classification tasks. Moreover, we derive a theoretical bound for the pure self-attention rank decay when using Sinkhorn normalization and find that rank decays to one doubly exponentially with depth, a phenomenon that has already been shown for softmax.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC) |
| Cite as: | arXiv:2604.07925 [cs.LG] |
| (or arXiv:2604.07925v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.07925 arXiv-issued DOI via DataCite |
|
| Journal reference: | Transactions on Machine Learning Research (TMLR), 2026, https://openreview.net/forum?id=fGItYoS8j1 |
Submission history
From: Michela Lapenna [view email]
[v1]
Thu, 9 Apr 2026 07:46:18 UTC (739 KB)
[v2]
Tue, 6 Oct 2026 10:16:43 UTC (765 KB)
来源：arXiv cs.AI · arxiv.org