arXiv cs.AI· Atreya Vedantam, Nataraj Das, Chandrashekar Lakshminarayanan·· 3 天前AI 评分22
线性模型中的延迟泛化动力学研究
On the Dynamics of Delayed Generalization in Linear Models
AI 导读
研究发现，延迟泛化（训练准确率远早于测试准确率接近完美）可在使用梯度下降和指数尾损失训练的线性模型中、在线性可分数据上出现。作者将有限时间超平面动力学刻画为三个阶段：种群阶段、支持向量主导的遗忘阶段和泛化阶段，并通过对种群与支持向量分布的控制实验验证了该理论，还在 MNIST 上单输出神经元的深度模型中验证了预测的偏置行为。
正文 · 原文
Abstract:Delayed generalization, where training accuracy becomes nearly perfect substantially before test accuracy, has often been studied in the context of the depth and compositional structure of deep networks. In this paper, we ask whether delayed generalization can occur in simpler settings. We show that it can arise in a linear model trained with gradient descent and an exponential-tail loss on linearly separable data. We characterize the finite-time hyperplane dynamics into three phases: a population phase, a support vector dominated unlearning phase and a generalization phase. The convergence of the residual beyond the maximum-margin direction is sufficient for the unlearning phase to begin. In the support vector dominated phase, we characterize the hyperplane dynamics. The time scale in these dynamics causes delayed generalization. We validate our theory through controlled experiments varying population and support-vector distributions, and further verify the predicted bias behavior in a deep model with a single output neuron on MNIST. Our results provide a unified characterization of the asymptotic and finite-time dynamics underlying delayed generalization in linear models.
| Comments: | v2: substantially revised and improved version |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.08302 [cs.LG] |
| (or arXiv:2602.08302v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.08302 arXiv-issued DOI via DataCite |
Submission history
From: Atreya Vedantam [view email]
[v1]
Mon, 9 Feb 2026 06:16:43 UTC (12,044 KB)
[v2]
Sun, 11 Oct 2026 12:45:29 UTC (640 KB)
[v3]
Wed, 14 Oct 2026 05:57:15 UTC (640 KB)
来源：arXiv cs.AI · arxiv.org