arXiv cs.CL· Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun·· 14 小时前AI 评分44
RePlaid 证明连续扩散模型在语言建模上可与离散扩散竞争
Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
AI 导读
研究者将基于似然的连续扩散语言模型 Plaid 与现代离散 DLM 架构对齐，构建出 RePlaid，并首次为连续 DLM 建立可与离散 DLM 相比的 scaling law：与自回归模型的计算差距仅 20×，在更少参数下优于 Duo，并在过训练区间优于 MDLM。
正文 · 原文
Abstract:While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only $20\times$ compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of $22.1$ among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.
| Comments: | NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2605.18530 [cs.CL] |
| (or arXiv:2605.18530v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.18530 arXiv-issued DOI via DataCite |
Submission history
From: Zhihan Yang [view email]
[v1]
Mon, 18 May 2026 15:15:24 UTC (657 KB)
[v2]
Wed, 9 Sep 2026 21:20:41 UTC (658 KB)
[v3]
Sat, 24 Oct 2026 20:05:07 UTC (1,670 KB)
来源：arXiv cs.CL · arxiv.org