arXiv cs.CL· Durga Prasad Maram, Dhruvesh Patel, Tahira Naseem, Benjamin Rozonoyer, Gaurav Pandey, Ram\'on Fernandez Astudillo, Andrew McCallum·· 13 小时前AI 评分22
δMDM：带纠错机制的插入式序列生成模型
Birth, Death and Diff: Insertion-Based Sequence Generation with Corrections
AI 导读
δMDM 为插入式掩码扩散模型加入替换与删除操作，使中间生成结果可被修正，从而支持更激进的并行解码。在 OpenWebText 无条件文本生成上，最高推理预算下生成困惑度较 FlexMDM 减半；在 TinyGSM 条件生成上以一半的 NFEs 达到 FlexMDM 同等准确率。该模型在速度—质量权衡上 Pareto 优于 FlexMDM。
正文 · 原文
Abstract:Insertion-based masked diffusion models enable flexible-length, parallel generation by interleaving mask insertion with token unmasking. However, because their generation decisions are irreversible, aggressive parallel decoding can introduce errors that subsequent steps cannot correct, necessitating conservative decoding with more forward passes (NFEs). Moreover, they exhibit higher entropy than standard MDMs because they marginalize over multiple alignments during training and therefore require even more conservative decoding in the absence of a correction mechanism. To address this limitation, we introduce ${\delta}$MDM, which makes intermediate generations revisable by equipping an insertion-based model with substitution and deletion operations. During training, we expose the model to errors caused by aggressive unmasking and insertion, teaching it to correct its own mistakes. At inference time, we interleave insertion and unmasking with correction steps that can revise previously unmasked tokens and remove unnecessary masks. This mechanism makes aggressive decoding practical, improving both generation speed and quality. ${\delta}$MDM Pareto-dominates FlexMDM, a vanilla insertion-based MDM, on the speed--quality trade-off. On high-diversity unconditional text generation on OpenWebText, it halves the generative perplexity relative to FlexMDM at the highest inference budget. It also achieves lower generative perplexity at all inference budgets compared to MDM trained from scratch with variable-length padded sequences. On low-diversity conditional text generation on TinyGSM, it matches FlexMDM accuracy with half as many NFEs.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34310 [cs.LG] |
| (or arXiv:2610.34310v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34310 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Durga Prasad Maram [view email]
[v1]
Sun, 25 Oct 2026 14:23:38 UTC (3,084 KB)
来源：arXiv cs.CL · arxiv.org