arXiv cs.AI· Jihwan Moon, Sheir A. Zaheer, Jinmyoung Lee, Gunhee Kim, Chan Y. Park·· 14 小时前AI 评分31
贪心局部学习用于语言模型预训练：差距与目标设计
Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design
AI 导读
一项在 125M 和 400M 参数、K∈{1,2,4} 分块、Chinchilla 最优预算下进行的 token 预算匹配实证研究发现，局部学习与端到端训练的损失差距从 K=2 到 K=4 扩大一倍以上，但 K=4 时从 125M 到 400M 会缩小。
正文 · 原文
Abstract:Greedy block-wise local learning splits a network into gradient-isolated blocks trained by local auxiliary losses, deleting the backward pass between blocks: inter-stage communication becomes forward-only and every block can step its optimizer independently, properties directly relevant to decentralized model-parallel training. Local learning is competitive with end-to-end backpropagation on image classification, and on small Transformers it is known to trade a worse best loss for parallel speedup. How this loss gap behaves in autoregressive language model (LM) pretraining at larger scale, and which auxiliary designs reduce it, has not been measured. We present a token-budget-matched empirical study at 125M and 400M parameters with $K \in \{1,2,4\}$ blocks at Chinchilla-optimal budgets, factorizing the auxiliary design into network architecture and training objective. We observe: (i) the gap to end-to-end training more than doubles from $K=2$ to $K=4$, but at $K=4$ shrinks from 125M to 400M; (ii) replacing an MLP auxiliary with a Transformer-based one is a strong network-side intervention, recovering 22-41% of the gap; (iii) a multi-token-prediction (MTP) auxiliary objective helps at the first block boundary, whereas adding it at deeper boundaries hurts, and restricting it to the first block yields the best $K=4$ configuration ($+0.062$ vs. $+0.075$ nats at 400M); and (iv) deployment-style per-block execution reduces activation memory by up to $2.2\times$. We frame these results as an empirically grounded method direction rather than a finalized method: local objectives should apply future-predictive pressure selectively across boundaries while resisting shortcuts that bypass predictive content.
| Comments: | Accepted at the NeurIPS 2026 Workshop on Collaborative, Open, and Decentralized Training of Foundation Models (CODEC-FM). 13 pages, 3 figures, 8 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.04867 [cs.LG] |
| (or arXiv:2610.04867v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.04867 arXiv-issued DOI via DataCite |
Submission history
From: Jihwan Moon [view email]
[v1]
Sun, 4 Oct 2026 02:10:52 UTC (157 KB)
[v2]
Mon, 26 Oct 2026 05:00:35 UTC (156 KB)
来源：arXiv cs.AI · arxiv.org