arXiv cs.CL· Parul Gupta, Satwik, Avni Aggarwal, Bhavesh Gurnani, Arnab Kumar Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla·· 13 小时前AI 评分31
HMDLM：分层掩码扩散语言模型
Hierarchical Masked Diffusion Language Models
AI 导读
研究者提出分层掩码扩散语言模型（HMDLM），为每个 token 位置附加训练数据中已有的层级信息，将生成拆成两遍：先预测掩码位置的层级分布，再据此预测 token 分布，并让各层级拥有独立噪声调度，从而在完全并行解码下优先揭示高优先级位置。
正文 · 原文
Abstract:Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation. However, standard masked diffusion language models use a shared noise schedule across positions without explicitly encoding which positions should be resolved first. Many tasks exhibit a natural priority among positions: document titles and section headings can guide the generation of body text, while Sudoku cells determined directly from the initial clues can help resolve more difficult cells. To incorporate this notion of priority (or hierarchy) among token positions, we introduce the Hierarchical Masked Diffusion Language Model (HMDLM), which augments each token position with a hierarchy level available as part of the training data, and requires zero or minimal additional annotations. It factors generation into two passes: the first predicts a hierarchy distribution over masked positions, the second predicts token distributions conditioned on it. Giving each hierarchy level its own noise schedule induces a soft, controllable preference for revealing higher-priority positions earlier while keeping decoding fully parallel. Our formulation is mathematically grounded, and we re-derive the forward and reverse processes and the NELBO for this factorization. Our evaluation shows that for the task of language generation on LM1B, HMDLM reduces perplexity by more than 1.5 points over Anchored Diffusion Language Model (ADLM), a recently proposed strong diffusion baseline; on WikiHow, we see a drop in perplexity of up to 4.4 pts on ADLM and up to 7 pts against MDLM. An LLM judge on WikiHow prefers HMDLM generations (the two variants combined) more than $2\times$ as often as the baseline. On the task of solving Sudoku puzzles requiring multi-step reasoning, it improves solution accuracy by up to $8.6\%$ absolute points over existing diffusion baselines.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.29255 [cs.CL] |
| (or arXiv:2610.29255v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29255 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Parul Gupta [view email]
[v1]
Wed, 21 Oct 2026 06:06:01 UTC (978 KB)
来源：arXiv cs.CL · arxiv.org