跳到正文
arXiv cs.AI· Jameson Sandler, Shane Bergsma, Mostafa Elhoushi, Bin Claire Zhang, Gavia Gray, Joel Hestness, Nolan Dey, Ferdinando Fioretto·· 3 小时前AI 评分32

DNeo:面向并行草稿模型的目标特征编码新架构

DNeo: Your Parallel Draft Model Needs Better Target Feature Encoding

AI 导读

DNeo 通过 Mixture-of-Features Attention(MoFA)等改进,实现比 SoTA(DFlash-2)快 1.3 倍的生成速度,且不增加草稿模型体积、不损失生成质量。研究指出标准特征编码存在有损、无记忆、目标层采样均匀三大局限,MoFA 以工作记忆编码此前被拒绝的提案,并结合 2-tap 因果卷积、仅前缀交叉注意力与非均匀目标层选择。

正文

View PDF HTML (experimental)

Abstract:Speculative-diffusion decoding uses a small parallel drafter model to generate b draft tokens in one forward pass, which then undergo parallel verification via a larger target model. Faster inference can be achieved when multiple draft tokens pass verification simultaneously. Recent parallel draft architectures (DFlash, DSpark) improve draft quality by conditioning on rich features computed by the target, through a mechanism known as feature encoding. However, standard feature encoding mechanisms have three limitations: (1) they are lossy, applying encoding projections that are destructive to the underlying target feature-space via a discovered effect known as residual mixing, (2) they are memory-less, with no record of previously processed proposals, and (3) they utilize features sampled uniformly from across the target layers, omitting important hidden states. To address these issues, we propose Mixture-of-Features Attention (MoFA), a lightweight feature-encoding approach equipped with a working memory that encodes previously rejected proposals. Leveraging MoFA, rejection memory, and a set of further enhancements, including 2-tap causal convolutions, prefix-only cross-attention, and non-uniform target layer selection, the final architecture, DNeo, achieves 1.3x faster generation speed than SoTA (DFlash-2) without impacting drafter footprint or generation quality. DNeo marks a new state of the art in high quality, fast LLM inference.
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2611.21925 [cs.AI]
  (or arXiv:2611.21925v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.21925

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jameson Sandler [view email]
[v1] Mon, 16 Nov 2026 21:41:20 UTC (1,903 KB)

来源:arXiv cs.AI · arxiv.org