arXiv cs.CL· Sneha Murthy Ghantasala·· 12 小时前AI 评分36
TreeWY：为 Gated DeltaNet 混合模型提供投机解码验证
TreeWY: Speculative Verification for Gated DeltaNet Hybrids
AI 导读
TreeWY 用树结构 WY 变换替代 GDN 层逐草稿位置的循环状态快照，只需一次三角求解算出所有草稿节点输出，提交时仅重建被接受的那一个状态。在 Qwen3.5 35B 和 397B 上的服务基准显示，该方法在接受长度基本不变的情况下降低了投机解码的循环状态内存与 KV-cache 压力，把释放的 HBM 转化为更高吞吐和更低 TTFT。更宽的草稿树因此变得可行，但尚未带来吞吐收益。
正文 · 原文
Abstract:Modern open models are hybrids: most layers are linear-attention (Gated DeltaNet, GDN) layers carrying a small fixed-size recurrent state instead of a growing key-value (KV) cache. This makes ordinary decoding memory-efficient, but hurts speculative decoding. To verify a batch of draft tokens and then roll back the rejected ones, today's systems snapshot the full recurrent state at every draft position for GDN layers, and those snapshots cannot be shared across branches of a draft tree, so a wide, high-acceptance tree becomes memory-infeasible. We remove the snapshots. Using a tree-structured WY transform of the gated delta rule, we compute every draft node's output with a single triangular solve and reconstruct only the one accepted state on commit, storing a small pseudo-value matrix instead of per-node states; the derivation depends only on the gated delta rule, not on any other architectural detail. In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B) this cuts speculative recurrent-state memory and KV-cache pressure at essentially identical acceptance length, turning the freed HBM into higher throughput and much lower time-to-first-token (TTFT) wherever memory binds, and costing a few percent where it does not. For tree width the same memory buys affordability: a wider, higher-acceptance draft becomes possible, though not yet a throughput win.
| Comments: | 10 pages, 3 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Performance (cs.PF) |
| Cite as: | arXiv:2608.20961 [cs.AI] |
| (or arXiv:2608.20961v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.20961 arXiv-issued DOI via DataCite |
Submission history
From: Sneha Murthy Ghantasala [view email]
[v1]
Fri, 21 Aug 2026 10:31:15 UTC (65 KB)
[v2]
Thu, 22 Oct 2026 18:32:01 UTC (57 KB)
来源：arXiv cs.CL · arxiv.org