跳到正文
arXiv cs.AI· Yongding Tian, Zaid Al-Ars, Maksim Kitsak, Peter Hofstee·· 4 小时前AI 评分31

低损失区域错位导致 Grokking:基于模式连通性的分析框架

Misalignment of Low-Loss Regions Causes Grokking

AI 导读

研究提出基于模式连通性与低损失区域几何的分析框架,指出 Grokking 源于训练集与验证集所诱导的低损失区域错位。在保持对称性的训练/验证划分下,标准模运算设置出现验证性能不恢复的稳定反例(anti-grokking),挑战了现有相关性解释。一旦两个区域良好对齐,仅靠训练超参数无法产生 Grokking,训练动态退化为可训练或不可训练两种行为。

正文 · 原文

View PDF HTML (experimental)

Abstract:Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism remains unsettled. In this work, we develop an analysis framework based on mode connectivity and the geometry of low-loss regions. The framework predicts that the standard modular-arithmetic setting does not always produce grokking: under a symmetry-preserving train/validation split, we observe a stable anti-grokking case in which validation performance does not recover. This counterexample challenges several existing correlational explanations of grokking. More broadly, our analysis framework and results further suggest that grokking arises when the low-loss regions induced by the training and validation partitions are misaligned. Once these regions become well aligned, training hyperparameters alone cannot produce grokking and the observed dynamics collapse to either trainable or non-trainable behavior.
Comments: 28 pages, 27 figures
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2610.00620 [cs.LG]
  (or arXiv:2610.00620v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2610.00620

arXiv-issued DOI via DataCite

Submission history

From: Yongding Tian [view email]
[v1] Wed, 30 Sep 2026 19:22:20 UTC (2,371 KB)
[v2] Tue, 17 Nov 2026 16:09:05 UTC (3,126 KB)

来源:arXiv cs.AI · arxiv.org