arXiv cs.AI· Patrick Krauss, Achim Schilling, Andreas Maier, Thomas Kinfe, Claus Metzner·· 10 小时前AI 评分31
神经网络权重空间与内部表征中留存可解码的学习轨迹痕迹
Learning Trajectories Leave Decodable Traces in Neural Network Weight Space and Internal Representations
AI 导读
训练后的神经网络权重保留了初始化、任务顺序与早期轨迹状态的选择性历史信息:30 个训练网络中全部正确匹配到各自初始权重,线性探针在 80 个留出网络上以 100% 准确率解码两个早期任务的顺序。初始权重本身无法被可靠重建,扰动网络恢复分类性能但不回到原训练路径;2,160 对对齐端点中有 2,157 对在采样插值点上无正损失障碍,表明不同解之间存在低损失连接。
正文 · 原文
Abstract:Neural networks trained on the same task develop related weight-space structures, yet remain distinguishable after learning. Do these differences merely reflect alternative solutions, or do they preserve information about individual learning histories? We tested whether final weights reveal initialization, task order, and earlier trajectory states. Training reorganized the ensemble: initial pairwise distances did not consistently predict final distances. Nevertheless, all 30 trained networks were correctly matched to their initial weights among ten candidates per dataset. Networks sharing an initialization also developed recognizable final hidden representations across different batch schedules, although direct comparisons with initial activity did not recover identity. Learning order remained readable as well. A linear probe decoded the order of two earlier tasks with 100% accuracy on 80 held-out networks after ten epochs of a common final task. Earlier intermediate weight states were partly predictable, but individual initial weights could not be reliably reconstructed. Final weights thus retain selective historical information rather than a complete trajectory record. Perturbed networks recovered classification performance without returning to their unperturbed training paths. This separation persisted across datasets, intervention times, and network layers. Moreover, 2,157 of 2,160 aligned endpoint pairs showed no positive loss barrier at sampled interpolation points, indicating low-loss connections between distinct solutions rather than convergence to one weight vector. Together, these findings show that a shared task constrains networks to related successful solutions without erasing their individual histories. Final weights reflect not only what a network has learned, but also where learning began and how it unfolded.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neurons and Cognition (q-bio.NC) |
| Cite as: | arXiv:2611.24782 [cs.LG] |
| (or arXiv:2611.24782v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24782 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Patrick Krauss [view email]
[v1]
Wed, 18 Nov 2026 17:27:55 UTC (5,221 KB)
来源:arXiv cs.AI · arxiv.org