arXiv cs.AI· Min Chen, Junqi Wang, Zeguan Wu, Wenhao Huang, Junyu Liu·· 2 天前AI 评分31
神经网络中的标度与重整化：用有效重整化群描述 Grokking 现象
Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking
AI 导读
研究用标度与重整化群（RG）理论定量描述神经网络中的 grokking 现象，即从记忆到泛化的延迟转变。针对两层网络的模加法任务，作者通过损失对每神经元范数的二阶展开构造精确耦合矩阵，并用其 Fiedler 向量定义有效几何，进而执行实空间 RG 块自旋聚合。线性化局部 RG 映射得到相关标度指数与五个导出临界指数，经验估计与 RG 预测吻合在 1-10% 以内。
正文 · 原文
Abstract:Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.
| Comments: | 22 pages, 10 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); High Energy Physics - Theory (hep-th); Quantum Physics (quant-ph) |
| Cite as: | arXiv:2610.36113 [cs.LG] |
| (or arXiv:2610.36113v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36113 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Min Chen [view email]
[v1]
Mon, 26 Oct 2026 18:25:08 UTC (3,423 KB)
来源：arXiv cs.AI · arxiv.org