arXiv cs.AI· Laura Gomezjurado Gonzalez·· 15 小时前AI 评分31
数字表示如何影响 Transformer 的算术泛化：Collatz 预测研究
The Long Delay to Arithmetic Generalization
AI 导读
研究发现 Transformer 在一步 Collatz 预测中的泛化延迟（grokking）只在训练集整数种类很少时出现，而数字的进制表示决定了需要多少整数才能泛化。当进位只依赖当前位、输出位仅依赖少数相邻输入位时，所需训练整数可减少数倍，且该顺序可在训练前预测。线性探针从随机初始化编码器中恢复输入低位残差的准确率与训练后相当，说明高探针准确率不能可靠证明模型学会了计算这些特征。
正文 · 原文
Abstract:Large language models today still make mistakes on multi-digit arithmetic when they do not have tools, and part of the reason is how numbers are split into tokens. Arithmetic is also a setting where the correct answer to every input and the algorithm that produces it are already known, so training transformers on it lets us study what they learn against a ground truth. These models often fit their training set long before they generalize to held-out inputs, and this delay, known as grokking, decreases as the training set grows. But, how does the representation of numbers change the amount of data a transformer needs to generalize? We study one-step Collatz prediction, in which a transformer reads the digits of an integer in a chosen base and outputs the digits of the next term in its Collatz sequence. We find that the long delay appears only when the training set has few different integers, and that the numeral base determines how many integers are sufficient. In bases where each output digit depends only on a few neighbouring input digits, because the carry between positions depends only on the current digit, a transformer needs several times fewer training integers. We predicted this ordering before training, and input length does not explain it. The same carry structure predicts which bases are easy for a second Collatz map, for integer division, for a decoder-only transformer, and across three model sizes. Finally, a linear probe recovers the low-order residues of the input as accurately from a randomly initialized encoder as from a trained one. Thus, high probe accuracy on such features does not reliably show that a model has learned to compute them. More broadly, we show that the choice of numeral representation affects sample efficiency in ways we can predict from the structure of the task before training.
| Comments: | 20 pages, 8 figures, 18 tables. Accepted at the NeurIPS 2026 Workshop on Interpretability as a Science. Revised and extended after peer review, with interpretations updated in light of additional experiments |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.13082 [cs.LG] |
| (or arXiv:2604.13082v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.13082 arXiv-issued DOI via DataCite |
Submission history
From: Laura Gomezjurado Gonzalez [view email]
[v1]
Mon, 30 Mar 2026 22:20:38 UTC (175 KB)
[v2]
Wed, 17 Jun 2026 04:09:14 UTC (301 KB)
[v3]
Fri, 23 Oct 2026 18:53:37 UTC (296 KB)
来源：arXiv cs.AI · arxiv.org