arXiv cs.AI· Hans Gundlach, Alex Fogelson, Jayson Lynch, Ana Trisovic, Jonathan Rosenfeld, Anmol Sandhu, Neil Thompson·· 10 小时前AI 评分58
研究：语言模型预训练中的算法效率提升与算力规模相关
On the Origin of Algorithmic Progress in AI: Evidence from Language Model Pre-Training
AI 导读
arXiv 论文对 2012 至 2023 年 AI 预训练算法效率提升 22,000 倍的说法做消融实验，在小算力规模下只能解释不到 10 倍，加上文献中其他创新也不到 100 倍。作者通过 LSTM 与 Transformer 的缩放实验发现两者在计算最优缩放律上存在指数差异，认为算法效率增益与算力规模绑定，小算力下的算法进展远慢于此前假设。该论文已被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Algorithms have been estimated to increase AI pretraining FLOP efficiency by a factor of 22,000 between 2012 and 2023 (Ho et al., 2024). Running ablation experiments on key innovations from this time period, we are able to account for less than 10x of these gains at small compute scales. Surveying the broader literature, we estimate that additional innovations not included in our ablations also account for less than 10x, yielding less than 100x efficiency gains total in low-compute regimes. This leads us to conduct scaling experiments, which provide evidence that the efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding evidence for exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that, contrary to standard assumptions, an algorithm's efficiency gains are tied to compute scale. Using experimental scaling extrapolations and literature estimates, we can account for the majority of efficiency gains over the same time period. Our results indicate that algorithmic progress at smaller training compute levels has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.
| Comments: | Accepted at NeurIPS 2026 (camera-ready version) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| MSC classes: | 68T07, 68Q25 |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2511.21622 [cs.LG] |
| (or arXiv:2511.21622v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2511.21622 arXiv-issued DOI via DataCite |
Submission history
From: Hans Gundlach [view email]
[v1]
Wed, 26 Nov 2025 17:46:31 UTC (5,021 KB)
[v2]
Tue, 10 Nov 2026 10:05:28 UTC (5,170 KB)
来源：arXiv cs.AI · arxiv.org