arXiv cs.CL· Giries Abu Ayoub, Mario Barbara, Llu\'is Pastor-P\'erez, Tanja Bien, Aneesh Barthakur, Alaa Maalouf, Loay Mualem·· 10 小时前AI 评分26
AXON:为扩散语言模型加速解码的免训练支持性 Token 揭示模块
Supportive Token Revealing for Fast Diffusion Language Model Decoding
AI 导读
AXON 是一个免训练模块,可叠加在现有扩散语言模型并行解码策略之上,通过注意力、不确定性和置信度信号选出“锚点”Token,把揭示标准从“哪些 Token 最安全”转为“哪些高置信揭示最能支持后续去噪”。在多个扩散语言模型的推理与代码生成基准上,AXON 改善了现有并行解码器的质量-延迟权衡,常能在保持或提升准确率的同时减少函数评估次数。
正文 · 原文
Abstract:Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.04236 [cs.CL] |
| (or arXiv:2606.04236v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.04236 arXiv-issued DOI via DataCite |
Submission history
From: Giries Abu Ayoub [view email]
[v1]
Tue, 2 Jun 2026 21:35:54 UTC (1,765 KB)
[v2]
Thu, 19 Nov 2026 05:04:09 UTC (1,771 KB)
来源:arXiv cs.CL · arxiv.org