arXiv cs.AI· Fares B. Mehouachi, Saif Eddin Jabari·· 12 小时前AI 评分22
用无噪声 l^p 范数控制缓解快速对抗训练中的灾难性过拟合
Catastrophic Overfitting, Entropy Gap and Participation Ratio: A Noiseless $l^p$ Norm Solution for Fast Adversarial Training
AI 导读
研究提出一种仅通过控制 l^p 训练范数来缓解灾难性过拟合（CO）的方法，无需噪声注入、正则化或梯度裁剪。作者发现 CO 在 l^∞ 范数下比 l^2 范数下更普遍，并用 Participation Ratio 与熵量化梯度集中度，据此设计自适应 l^p-FGSM 自动调节训练范数。实验显示该方法取得有竞争力的鲁棒性，代码已公开。
正文 · 原文
Abstract:Adversarial training is a cornerstone of robust deep learning, but fast methods like the Fast Gradient Sign Method (FGSM) often suffer from Catastrophic Overfitting (CO), where models become robust to single-step attacks but fail against multi-step variants. While existing solutions rely on noise injection, regularization, or gradient clipping, we propose a novel solution that purely controls the $l^p$ training norm to mitigate CO. Our study is motivated by the empirical observation that CO is more prevalent under the $l^{\infty}$ norm than the $l^2$ norm. Leveraging this insight, we develop a framework for generalized $l^p$ attack as a fixed point problem and craft $l^p$-FGSM attacks to understand the transition mechanics from $l^2$ to $l^{\infty}$. This leads to our core insight: CO emerges when highly concentrated gradients, where information localizes in few dimensions, interact with aggressive norm constraints. By quantifying gradient concentration through Participation Ratio and entropy measures, we develop an adaptive $l^p$-FGSM that automatically tunes the training norm based on gradient information. Extensive experiments demonstrate that this approach achieves competitive robustness without requiring additional regularization or noise injection, providing a novel and theoretically-principled pathway to mitigate the CO problem.
| Comments: | 25 pages, 16 figures, 7 tables. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2505.02360 [cs.LG] |
| (or arXiv:2505.02360v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.02360 arXiv-issued DOI via DataCite |
Submission history
From: Fares B. Mehouachi [view email]
[v1]
Mon, 5 May 2025 04:41:21 UTC (4,666 KB)
[v2]
Sat, 16 May 2026 12:36:31 UTC (4,667 KB)
[v3]
Fri, 23 Oct 2026 11:28:52 UTC (3,413 KB)
来源：arXiv cs.AI · arxiv.org