arXiv cs.CL· Haoning Yang, Lili Pan·· 13 小时前AI 评分38
GRAFT：为连续扩散语言模型补回缺失的采样接口
GRAFT: Restoring the Missing Sampling Interface in Continuous Diffusion Language Models
AI 导读
连续扩散语言模型 ELF 在常见词上与人类文本几乎一致，但最稀有词出现率仅为人类的 0.30x，且从 105M 到 652M 参数都未收窄。研究将其归因于缺少逐步离散采样、每步对候选做平均，并提出免训练方法 GRAFT：在模型未定夺时锁定稀有词再整体重生成。
正文 · 原文
Abstract:Continuous diffusion language models generate a whole sequence in parallel, never drawing a single word at any intermediate step; only the final step turns embeddings into text. Models such as ELF now report autoregressive-level quality at a fraction of the sampling steps of discrete-token diffusion models, and this design principle -- no step-wise discretization -- is what their own ablations credit for the gain. We show that this same principle carries a hidden cost: a collapse of the lexical long tail. Measured against held-out human text, ELF's output matches the human rate on common words almost exactly, yet its rarest words appear at only 0.30x the human rate, a gap that does not close from 105M to 652M parameters and that no sampler setting these models expose can reach. A same-size GPT-2, in contrast, spans zero to above the human rate by its decoding rule alone. We trace the difference to what continuous diffusion models lack: with no step that ever draws a discrete word, every step is instead driven by an average over the candidates still in play, and averaging a rare candidate together with the common word competing for its slot lands on the common word. An autoregressive model avoids this because its decoding rule samples one candidate rather than averaging over all of them. We propose GRAFT, a training-free method that grafts this missing choice back onto a frozen model: while the model is still undecided, GRAFT commits the rare words the average would otherwise discard, then regenerates the sequence from scratch with those commitments held in place. Across three ELF scales and on LangFlow, an independent continuous dLLM, GRAFT restores the rare-word share of generated text from 0.42-0.77x the human rate to 0.82-1.10x, a relative gain of 34-109%, and moves the entire word-frequency profile closer to human on every model tested, by 2.7x on ELF-B and 12x on LangFlow.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.29136 [cs.CL] |
| (or arXiv:2610.29136v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29136 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haoning Yang [view email]
[v1]
Wed, 21 Oct 2026 03:15:30 UTC (242 KB)
来源：arXiv cs.CL · arxiv.org