arXiv cs.AI· Guy Smorodinsky, Sveta Gimpleson, Itay Safran·· 6 天前AI 评分22
非线性神经网络中 GD 的收敛速率：对抗鲁棒性视角
On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspective
AI 导读
研究证明，即便在仅含两个神经元 ReLU 网络和两个训练样本的极简二分类设定下，GD 虽能收敛到最优鲁棒性间隔，但收敛速率严格为 Θ(1/ln(t))，极为缓慢。这是首个针对非线性模型鲁棒性间隔收敛速率的显式下界，实证模拟显示该失效模式在多种自然网络初始化下普遍存在且速率一致。
正文 · 原文
Abstract:We study the convergence dynamics of Gradient Descent (GD) in a minimal binary classification setting, consisting of a two-neuron ReLU network and two training instances. We prove that even under these strong simplifying assumptions, while GD successfully converges to an optimal robustness margin, effectively maximizing the distance between the decision boundary and the training points, this convergence occurs at a prohibitively slow rate, scaling strictly as $\Theta(1/\ln(t))$. To the best of our knowledge, this establishes the first explicit lower bound on the convergence rate of the robustness margin in a non-linear model. Through empirical simulations, we further demonstrate that this inherent failure mode is pervasive, exhibiting the exact same tight convergence rate across multiple natural network initializations. Our theoretical guarantees are derived via a rigorous analysis of the GD trajectories across the distinct activation patterns of the model. Specifically, we develop tight control over the system's dynamics to bound the trajectory of the decision boundary, overcoming the primary technical challenge introduced by the non-linear nature of the architecture.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.02095 [cs.LG] |
| (or arXiv:2603.02095v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.02095 arXiv-issued DOI via DataCite |
Submission history
From: Guy Smorodinsky [view email]
[v1]
Mon, 2 Mar 2026 17:13:33 UTC (38 KB)
[v2]
Thu, 8 Oct 2026 18:50:56 UTC (38 KB)
来源：arXiv cs.AI · arxiv.org