SpikeGPT 类脉冲语言模型的首个 Chinchilla 式 IsoFLOP 缩放定律研究

View PDF HTML (experimental)

Abstract:Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale. We present the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from $3\times10^{17}$ to $10^{19}$ FLOPs on FineWeb-EDU. We report two findings. First, the compute-optimal frontier is $N_{opt} \propto C^{0.56}$, close to the 0.50 that Hoffmann et al. report for dense Transformers, so spiking does not change how parameters and data should be traded off under a compute budget. Second, against non-spiking RWKV and Transformer baselines trained the same way, the loss gap splits into a backbone cost of about 0.16 nats that does not change with compute and a spiking cost that falls steadily with compute, from 0.34 to 0.23 nats over our range. These results give the first compute-optimal recipe for spiking language models and quantify the accuracy cost of their sparsity.

Comments:	Accepted to AXIOM: Foundations of Efficient Deep Learning (NeurIPS 2026 workshop). 4 pages + appendix, 4 figures, 5 tables. v2: corrected spiking cost at 1e18 and 3e18 FLOPs (spiking and baseline runs had been scored on different validation splits); the spiking cost now falls steadily rather than levelling off
Subjects:	Machine Learning (cs.LG); Computation and Language (cs.CL); Neural and Evolutionary Computing (cs.NE)
Cite as:	arXiv:2610.15146 [cs.LG]
	(or arXiv:2610.15146v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.15146 arXiv-issued DOI via DataCite

Submission history

From: Daniel Saunders [view email]
[v1] Sat, 10 Oct 2026 19:46:24 UTC (1,097 KB)
[v2] Tue, 20 Oct 2026 15:59:21 UTC (1,096 KB)