arXiv cs.AI· Yanyang Xiao·· 2 天前AI 评分22
浅层神经网络为何先学正弦分量：一个正交多项式动力学框架
Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework
AI 导读
该研究用正交多项式基将网络输出展开为广义傅里叶级数，把梯度下降动力学写成常微分方程，通过分析动力学矩阵 H 的特征值衰减与特征向量形状来解释频谱偏置。该框架可推广到 sigmoid、ReLU 等一般激活函数，对数据分布假设较弱，并避免有限域傅里叶分析中的混叠效应。
正文 · 原文
Abstract:Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.
| Comments: | 15 pages, 0 figures. The main draft dates back to 2019, with a new abstract and introduction section |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Spectral Theory (math.SP); Machine Learning (stat.ML) |
| MSC classes: | 68T07 |
| Cite as: | arXiv:2610.21964 [cs.LG] |
| (or arXiv:2610.21964v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21964 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yanyang Xiao [view email]
[v1]
Thu, 15 Oct 2026 14:23:15 UTC (40 KB)
来源：arXiv cs.AI · arxiv.org