arXiv cs.AI· Zhaohui Zheng, Chenhang He, Jinrui Zhang, Ming-Ming Cheng, Lei Zhang·· 3 天前AI 评分29
HaloBlock：为扩散大语言模型带来更快更好的解码
HaloBlock: Faster and Better Decoding for Diffusion Large Language Models
AI 导读
HaloBlock 是一种面向扩散大语言模型（dLLM）推理的难度感知自适应 halo 机制，通过双向累积熵刻画块解码难度，并在每个块两侧自适应分配双向 halo 区域以缓解边界上下文不连续。在 LLaDA-8B-Instruct、Dream-7B-Instruct 和 LLaDA-1.5 上针对数学推理与代码生成基准的评测显示，HaloBlock 在准确率与推理效率上均有一致提升，源代码已公开。
正文 · 原文
Abstract:Diffusion large language models (dLLMs) have attracted increasing research attention due to their bidirectional attention and arbitrary order decoding. However, there are two issues in dLLMs inference: the decoding difficulty is dominated by the tokens on the right side of the block, and the block-wise decoding inherently suffers from boundary contextual discontinuity. To tackle these limitations, we propose HaloBlock, a difficulty-aware adaptive halo mechanism for dLLM inference. We first leverage bidirectional cumulative entropy to characterize block decoding difficulty. Then, based on the bidirectional decoding difficulty, we adaptively allocate bidirectional halo regions on both sides of each block to mitigate boundary contextual discontinuity. The difficulty-aware design of HaloBlock smooths cross-block context propagation and lowers decoding difficulty for hard tokens, yielding better generation quality and faster convergence. Evaluated on LLaDA-8B-Instruct, Dream-7B-Instruct and LLaDA-1.5 across mathematical reasoning and code generation benchmarks, HaloBlock demonstrates consistent accuracy improvements and higher inference efficiency. Source codes are publicly available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.17948 [cs.LG] |
| (or arXiv:2610.17948v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17948 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhaohui Zheng [view email]
[v1]
Tue, 13 Oct 2026 08:32:30 UTC (7,964 KB)
来源：arXiv cs.AI · arxiv.org