arXiv cs.CL· Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu·· 11 小时前AI 评分42
LangFlow：连续扩散模型在语言建模上媲美离散扩散
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
AI 导读
LangFlow 通过将嵌入空间扩散语言模型与 Flow Matching 结合，首次证明连续扩散在生成质量上可与离散扩散匹敌，在 LM1B 上生成困惑度达 92.3、OpenWebText 上达 36.5。
正文 · 原文
Abstract:Continuous diffusion has served as a foundation for high-fidelity, controllable, and few-step generation across continuous data modalities such as images, videos, and molecular structures. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts. Existing categorical and simplex-based approaches operate over extremely large and sparse language spaces, while prior embedding-space approaches avoid this sparsity but lack a well-selected design space. In this work, we close this gap with LangFlow by connecting embedding-space DLMs to Flow Matching, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, which improves both likelihood and sample quality for embedding-space DLMs and behaves differently from its use in discrete diffusion. Putting everything together, LangFlow is competitive with top discrete DLMs on generative quality, reaching a generative perplexity of 92.3 on LM1B and 36.5 on OpenWebText. It also outperforms discrete counterparts on reasoning tasks like Sudoku. As the first continuous DLM shown to rival discrete diffusion in generative quality, LangFlow provides clear evidence that continuous diffusion is a promising paradigm for language modeling. Code: this https URL.
| Comments: | NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.11748 [cs.CL] |
| (or arXiv:2604.11748v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.11748 arXiv-issued DOI via DataCite |
Submission history
From: Chumeng Liang [view email]
[v1]
Mon, 13 Apr 2026 17:21:41 UTC (491 KB)
[v2]
Tue, 14 Apr 2026 17:20:50 UTC (492 KB)
[v3]
Wed, 15 Apr 2026 16:15:30 UTC (493 KB)
[v4]
Thu, 5 Nov 2026 22:09:09 UTC (524 KB)
来源：arXiv cs.CL · arxiv.org