arXiv cs.CL· Naji Alazhar, Wassel Alazhar·· 6 天前AI 评分27
Adaptive DHiSparse：基于负载自适应 KV 层级结构的投机解码
Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy
AI 导读
Adaptive DHiSparse 在分层稀疏 KV 上支持投机解码，并按负载自适应调整驻留策略，保留 DSpark 与 DFlash2 的提议和接受规则。在 100K 上下文、并发 128 下，GLM-5.3 搭配 DFlash2 每解码 GPU 输出 793 tokens/s，为 HiSparse 的 1.45 倍；在相同延迟约束下可支撑最高 4 倍并发。
正文 · 原文
Abstract:Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45$\times$ HiSparse and 1.27$\times$ non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4$\times$ the concurrency and 2.3 to 3.7$\times$ the output, depending on the per-token bound. On DeepSeek-V4.1-Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85$\times$ the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.
| Comments: | 16 pages, 11 figures, 6 tables |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.13708 [cs.AI] |
| (or arXiv:2610.13708v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13708 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Naji Alazhar [view email]
[v1]
Fri, 9 Oct 2026 15:02:35 UTC (309 KB)
来源：arXiv cs.CL · arxiv.org