arXiv cs.AI· Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu·· 9 天前AI 评分40
Mooncake：面向 LLM 服务的以 KVCache 为中心的分离式架构
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
AI 导读
Moonshot AI 为 Kimi 打造的 Mooncake 服务架构，将预填充与解码集群分离，并利用 GPU 集群闲置的 CPU、DRAM 和 SSD 构建分离式 KVCache 缓存，核心是以 KVCache 为中心的调度器，在满足延迟 SLO 的同时最大化有效吞吐。
正文 · 原文
Abstract:Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests.
| Comments: | 23 pages, 13 figures |
| Subjects: | Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR) |
| Cite as: | arXiv:2407.00079 [cs.DC] |
| (or arXiv:2407.00079v5 [cs.DC] for this version) | |
| https://doi.org/10.48550/arXiv.2407.00079 arXiv-issued DOI via DataCite |
Submission history
From: Ruoyu Qin [view email]
[v1]
Mon, 24 Jun 2024 02:05:32 UTC (264 KB)
[v2]
Tue, 2 Jul 2024 02:49:35 UTC (264 KB)
[v3]
Tue, 9 Jul 2024 04:03:10 UTC (280 KB)
[v4]
Wed, 3 Sep 2025 14:56:29 UTC (245 KB)
[v5]
Thu, 8 Oct 2026 10:55:46 UTC (238 KB)
来源：arXiv cs.AI · arxiv.org