arXiv cs.AI· Yunji Wang, Junjie Yao, Linyu Liu, Pinyan Lu, Zhi-Qin John Xu·· 9 天前AI 评分21
概率签名动力学：两层网络如何学会模加法
Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks
AI 导读
研究用"概率签名"方法揭示两层网络学习模加法的机制：梯度交互由训练分布的循环移位算子表达，并被离散傅里叶变换对角化，从而近似解耦出各傅里叶模态动力学，解释了傅里叶稀疏性、频率匹配与相位对齐的出现。该框架还解释了标签噪声下受损样本早期损失下降快于干净样本的现象——噪声增加条件标签碰撞、强化早期共享坐标。以 XOR 为例的实验也观察到预测频率。
正文 · 原文
Abstract:Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization. While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear. We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution. For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics. This explains the emergence of Fourier sparsity, frequency matching, and phase alignment. The same framework resolves a puzzle under label noise: corrupted examples can show faster early loss decrease than clean examples, despite lacking a coherent generalization rule. We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement. Finally, this method can be applied to other operators. Taking XOR as an example, we observed the predicted frequency in experiments.
| Comments: | 36 pages, 18 figures. Submitted to ICLR 2027 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11833 [cs.AI] |
| (or arXiv:2610.11833v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11833 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yunji Wang [view email]
[v1]
Thu, 8 Oct 2026 12:27:35 UTC (760 KB)
来源：arXiv cs.AI · arxiv.org