arXiv cs.AI· Shuaidi Wang, Zhuocheng Gong, Xuejie Liu, Jianguo Li, Yu Zhang·· 10 小时前AI 评分38
LED:为扩散语言模型引入 Levenshtein 式插入与删除编辑
Beyond Replacement: Levenshtein Editing for Diffusion Language Models
AI 导读
研究者提出 LED(Levenshtein Editing for Diffusion Language Models)后训练框架,在替换式编辑(ROE)基础上加入插入与删除,实现序列级 Levenshtein 式修正。
正文 · 原文
Abstract:Diffusion language models (DLMs) offer a flexible alternative to autoregressive models, enabling any-order infilling. While most existing decoding methods treat decoded tokens as fixed, some recent models allow previously decoded tokens to be revised only by replacing them with alternative tokens, a paradigm we refer to as replacement-only editing (ROE). Although ROE can correct individual tokens, it is inefficient for sequence shifts caused by missing or extra tokens and struggles when errors occur across many positions. To address these limitations, we propose LED (Levenshtein Editing for Diffusion Language Models), a post-training framework that extends ROE with insertion and deletion, enabling Levenshtein-style refinement of generated sequences. It uses two training designs. First, it reconstructs edit supervision on the fly during supervised fine-tuning using longest-common-subsequence alignment, enabling insertion and deletion learning without manually annotated labels. Second, it adopts multi-stage training, where the state from one stage becomes the input to the next, exposing the model to its earlier errors and enabling it to correct them. When applied to the 16B-parameter LLaDA2.1-Mini Base, LED raises the average score across twelve benchmarks from 54.86 to 57.10 under confidence-guided decoding while increasing tokens per forward from 6.14 to 7.05, whereas under confidence-free decoding it raises the average score from 27.52 to 47.19. We further show that LED achieves higher pass@k at large k than thresholded decoding without editing, demonstrating improved generation diversity and revealing the diversity potential of DLMs without confidence guidance. LED also improves robustness to large block sizes, raising the average score from 36.97 to 51.34 at block size 128. Together, these results suggest that LED supports more robust self-correction in DLMs.
| Comments: | 21 pages, 9 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23750 [cs.CL] |
| (or arXiv:2611.23750v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23750 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuaidi Wang [view email]
[v1]
Wed, 18 Nov 2026 03:05:15 UTC (582 KB)
来源:arXiv cs.AI · arxiv.org