arXiv cs.AI· Shreyas Chaudhari, Jose M. F. Moura·· 10 小时前AI 评分18
ReLU 网络仅训练偏置时的动力学研究
Bias-Only Training Dynamics in ReLU Networks
AI 导读
研究揭示,单隐层 ReLU 网络在权重随机初始化且固定、仅训练偏置时,偏置梯度对每个输入的依赖只通过其激活编码体现,激活编码不变时训练等价于以编码为特征的线性最小二乘问题。训练因此可视为一系列最小二乘问题的序列,神经元激活状态改变时发生切换。网络越宽,切换神经元的比例越小,无限宽极限下动力学趋近于初始激活编码上的线性最小二乘,实验与理论一致。
正文 · 原文
Abstract:Neural networks with randomly initialized, fixed weights and trainable biases are universal approximators, yet the dynamics of bias-only training remain largely unexplored. We study these dynamics in one-hidden-layer ReLU networks through activation codes, the binary patterns of active neurons that partition the input space into convex polyhedral regions. We show that the gradients with respect to the biases depend on each input only through its activation code, and hence the training dynamics can be characterized with these codes. In particular, as long as the activation codes remain fixed, training reduces to a linear least-squares problem with the codes as features. Training can then be understood as a sequence of least-squares problems, with transitions occurring when neurons change their activation states. We show that the fraction of switching neurons vanishes as the network widens, and the training dynamics approach those of linear least squares on the initial activation codes in the infinite-width limit. Our experiments support these theoretical findings, showing that while activation switches can noticeably affect training in narrower networks, their relative frequency and the associated changes in the dynamics diminish as the width increases.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24797 [cs.LG] |
| (or arXiv:2611.24797v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24797 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shreyas Chaudhari [view email]
[v1]
Wed, 18 Nov 2026 17:41:18 UTC (113 KB)
来源:arXiv cs.AI · arxiv.org