跳到正文
arXiv cs.AI· Truong Xuan Khanh·· 10 小时前AI 评分26

研究:响应相关信息不足以判定神经网络训练状态等价

Response-Relevant Information Is Not Sufficient for Neural Training-State Equivalence

AI 导读

一项在单层 Transformer 学习模加法的实验提出,两个检查点只有在训练受扰动并继续后响应相似,才算相对于干预族和时域的训练状态等价;对权重衰减脉冲的响应高度可复现(500 步时 rho_state = 0.934)。15 种权重谱、表示几何与优化器几何描述符能预测留出响应偏差,但其在匹配检查点上 16.1% 的优势无法与保留各描述符对常规观测量拟合依赖的条件安慰剂区分开。

正文 · 原文

View PDF HTML (experimental)

Abstract:Response-relevant information is not sufficient evidence that two neural networks are in the same training state. We define training-state equivalence operationally: two checkpoints are equivalent, relative to an intervention family and horizon, if they respond alike when training is perturbed and continues. On fresh runs of a one-layer transformer learning modular addition, response to a weight-decay pulse is highly reproducible relative to between-checkpoint differences (rho_state = 0.934 at 500 steps), establishing the target as measurable. Fifteen descriptors of weight spectra, representation geometry and optimiser geometry carry within-run response-relevant information beyond weight norm and loss on a development dataset, predicting held-out response deviation across seed blocks. But their apparent 16.1% advantage at matching checkpoints by future response does not separate from a conditional placebo preserving each descriptor's fitted dependence on the conventional observables, and a calibrated positive control makes simple insensitivity an unlikely explanation. Information a decoder can extract need not be organised in the pairwise geometry that similarity judgments require.
Comments: 18 pages, 5 figures. Code and provenance verified artifacts: this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23245 [cs.LG]
  (or arXiv:2611.23245v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.23245

arXiv-issued DOI via DataCite

Submission history

From: Xuan Khanh Truong [view email]
[v1] Fri, 4 Sep 2026 05:49:16 UTC (111 KB)

来源:arXiv cs.AI · arxiv.org