arXiv cs.CL· Jinjin He, Xinsong Feng, Lucheng Fu, Sinan Wang, Bo Zhu, Jiayun Wang·· 6 天前AI 评分31
重新思考扩散与解码的角色分工：面向并行语言生成的 Predictive-State Flows
Rethinking the Roles of Diffusion and Decoding: Predictive-State Flows for Parallel Language Generation
AI 导读
研究者提出 Predictive-State Flows（PSF），将序列划分为块，由 rectified-flow 先验仅生成基于前文计算的预测状态，再由块因果解码器并行生成所有块，把扩散从 L 个 token 级状态降到 K << L 个预测状态，token 级串行深度降至 L/K。
正文 · 原文
Abstract:In continuous diffusion language models (DLMs), latents generated by the diffusion prior are mapped almost deterministically to tokens, either by a decoder or through a token embedding matrix. Unlike in image generation, where visual details can vary continuously, the diffusion prior must capture fine-grained discrete token choices in continuous space, which can make the joint latent distribution difficult to model. We argue that generative roles should instead be divided: the diffusion prior should generate high-level predictive information, while the decoder should resolve the remaining token-level choices. Based on this view, we introduce Predictive-State Flows (PSF), which partitions a sequence into blocks and associates each block with a predictive state computed solely from the preceding context. A rectified-flow prior generates the sequence of predictive states, after which a block-causal decoder generates all blocks in parallel, autoregressively only within each block. This design reduces diffusion from L token-level states to K << L predictive states and reduces token-level sequential depth to L/K. On unconditional OpenWebText generation, a 197M-parameter model with K=8 achieves a generative perplexity of 22.8 at entropy 5.16 using 0.31 TFLOPs per 1024-token document, versus 24.0 and 7.2 TFLOPs for ELF-B with 32 sampling steps. The corresponding measured operating points yield MAUVE@512 of 0.93 and 0.73, respectively. Controlled comparisons further show that content states, which additionally encode their own blocks, yield 8-16% higher generative perplexity than predictive states, and that replacing sampled predictive-state trajectories with a shared mean lowers MAUVE@512 by 0.11. These results suggest that latent design for continuous DLMs should consider not only how compactly text is represented, but also how generative roles are divided between the prior and the decoder.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12898 [cs.CL] |
| (or arXiv:2610.12898v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12898 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jinjin He [view email]
[v1]
Fri, 9 Oct 2026 02:43:30 UTC (221 KB)
来源：arXiv cs.CL · arxiv.org