离散扩散语言模型的机制引导干预：自适应调度实现不破坏质量的受控生成

View PDF HTML (experimental)

Abstract:Discrete diffusion language models (DLMs) generate text by iteratively denoising all positions in parallel, offering an alternative to autoregressive models. Controlled generation methods for DLMs, imported from autoregressive models, apply uniform intervention at every denoising step. We show this uniform schedule is inefficient and degrades quality, and the damage compounds when multiple attributes are steered jointly. To diagnose the failure, we train sparse autoencoders on four DLMs (124M-8B parameters) and find that different attributes commit on distinct schedules, varying in timing, sharpness, and magnitude. For instance, topic commits within the first 2% of denoising on MDLM, whereas sentiment emerges gradually over 20% of the process. Motivated by these profiles, we propose an adaptive scheduling mechanism that concentrates intervention where each attribute is actively forming. An idealized allocation analysis predicts that attributes with more sharply concentrated emergence benefit more from adaptive scheduling, a prediction we confirm empirically. Across seven single- and multi-attribute steering tasks on four DLMs, adaptive steering consistently improves the control-quality balance over uniform and interval-restricted baselines.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:	arXiv:2605.10971 [cs.LG]
	(or arXiv:2605.10971v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.10971 arXiv-issued DOI via DataCite

Submission history

From: Hanhan Zhou [view email]
[v1] Fri, 8 May 2026 18:52:17 UTC (4,221 KB)
[v2] Tue, 6 Oct 2026 20:54:30 UTC (4,337 KB)