arXiv cs.AI· Xuan Wang, Zhuanhao Wu, Tingzhu Fang, Xushi Li, Sicheng Zhang, Zhenyu Yang, Feng Liu, Zhaohui Fan, Qiang Xie, Huayi Zou, Yuncong Liu, Zeyu Song, Ke Zhu, Yanjia Zeng, Jingchun Gao, Yan Shi, Kui Luo, Gezi Li, Liangjun Feng, Cheng Li·· 12 小时前AI 评分38
FlashOffloading：面向长上下文 LLM 推理的低延迟 KV Cache 卸载方案
FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
AI 导读
FlashOffloading 是一种以主机 DRAM 为中心的推理架构，将 prefill 与 decode 阶段的 KV Cache 全部卸载到主机内存，通过滚动 HBM 缓存、decode 热缓存、反向稀疏索引和计算-加载重叠 kernel 降低卸载延迟。
正文 · 原文
Authors:Xuan Wang, Zhuanhao Wu, Tingzhu Fang, Xushi Li, Sicheng Zhang, Zhenyu Yang, Feng Liu, Zhaohui Fan, Qiang Xie, Huayi Zou, Yuncong Liu, Zeyu Song, Ke Zhu, Yanjia Zeng, Jingchun Gao, Yan Shi, Kui Luo, Gezi Li, Liangjun Feng, Cheng Li
Abstract:In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference easily hits the high-bandwidth memory (HBM) capacity wall and results in low concurrency. Offloading the KV cache to host dynamic random-access memory (DRAM) frees HBM space and raises concurrency, but the additional cache management and host-to-device (H2D) load overhead inflates inference latency, higher concurrency fails to translate into effective throughput. We present FlashOffloading, a host-DRAM-centric inference architecture that fully offloads the KV cache in both the prefill and decode phases. FlashOffloading builds rolling HBM cache for prefill, a hot cache for decode, a reverse sparse index with zero extra miss-lookup overhead, and a compute--load overlapped kernel, optimizing KV offloading latency across the entire pipeline and delivering throughput gains under a time-per-output-token (TPOT) service-level objective (SLO). In experiments deploying GLM-5.2 with multi-token prediction (MTP3) speculative decoding on an Ascend910C prefill--decode (PD)-disaggregated platform, FlashOffloading improves prefill throughput by up to 1.38\(\times\) and decode throughput by up to 1.91\(\times\) under a 30 ms TPOT constraint. FlashOffloading is integrated into an open-source serving framework: this https URL.
| Comments: | 13 pages including references,13 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07542 [cs.AI] |
| (or arXiv:2611.07542v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07542 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tingzhu Fang [view email]
[v1]
Thu, 5 Nov 2026 06:50:05 UTC (2,183 KB)
来源：arXiv cs.AI · arxiv.org