CoBit：用比特流扩散进行语言建模

View PDF

Abstract:We introduce CoBit, a continuous diffusion language model that generates text by jointly denoising fixed-width binary token codes. Grouping each code's bits into a single Transformer patch preserves sequence length while replacing vocabulary-wide predictions with $\lceil\log_2 V\rceil$ bit logits. A matched-filter residual parameterization supplies the analytic isolated-bit posterior, leaving the network to learn contextual corrections. Stochastic sampling on an entropy-rate grid provides the largest empirical gain; the grid allocates both solver resolution and effective Langevin correction to noise levels where information is resolved. At 256 denoiser evaluations, CoBit-S (130M parameters) achieves generative perplexity 59.76 on LM1B and 27.06 on OpenWebText, at unigram entropies 4.31 and 5.26, respectively. Scaling to 462M parameters improves OpenWebText generative perplexity to 19.48 at entropy 5.40. Beyond open-ended text generation, CoBit achieves 65.9% exact-match accuracy on hard Sudoku. On GSM8K, score-temperature scaling with an untempered final integration step raises CoBit-S accuracy from 28.1% to 35.3% without additional denoiser evaluations, placing it within the reported range of temperature-tuned discrete diffusion baselines. Matched-trunk profiling also shows lower peak memory and higher throughput than a dense token-output baseline.

Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2605.07013 [cs.CL]
	(or arXiv:2605.07013v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2605.07013 arXiv-issued DOI via DataCite

Submission history

From: Georgios Batzolis [view email]
[v1] Thu, 7 May 2026 22:50:51 UTC (2,269 KB)
[v2] Mon, 15 Jun 2026 15:19:18 UTC (2,632 KB)
[v3] Fri, 6 Nov 2026 16:59:55 UTC (2,480 KB)