arXiv cs.AI· Daniel T. Bernstein, Chase Goddard, Luca Di Carlo, Vudtiwat Ngampruetikorn, David J. Schwab·· 3 天前AI 评分22
Fisher 信息与神经网络记忆的几何结构
Fisher information and the geometry of memorization in neural networks
AI 导读
研究揭示 Fisher 信息可刻画神经网络如何在泛化与记忆之间分配容量。在 CIFAR-10 上训练的视觉模型中，将权重投影到 Fisher 特征向量顶部可解耦通用任务性能与噪声样本记忆，且该效应在不同层表现各异。相同现象在小 Transformer 的合成上下文学习任务及可解析线性模型中均得到复现。
正文 · 原文
Abstract:A fruitful approach towards understanding generalization in machine learning involves characterizing how models separate generalizable structure from idiosyncratic or even corrupted features of training data. Fisher information has emerged as a diagnostic tool for this separation, leveraging loss geometry to distinguish generalization from memorization. Here we investigate this connection in vision models trained on CIFAR-10 with controlled noise that can be learned only through memorization. We show that projecting model weights onto the top Fisher eigenvectors decouples general task performance from noisy sample memorization, and we investigate how this decoupling presents differentially across layers. We show that the same phenomena are present in small transformers trained to solve a synthetic in-context learning task, and are reproduced in closed form in a simple, analytically solvable linear model. Our results reveal that Fisher information captures aspects of the geometric structure underlying how neural networks allocate capacity between generalization and memorization.
| Comments: | 20 pages (10 pages main text), 11 figures (6 figures main text), NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16161 [cs.LG] |
| (or arXiv:2610.16161v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16161 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Daniel Bernstein [view email]
[v1]
Mon, 12 Oct 2026 04:34:54 UTC (2,177 KB)
来源：arXiv cs.AI · arxiv.org