arXiv cs.AI· Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese, Wesley Harris·· 2 天前AI 评分31
WAKMan：负载感知的 KVCache 管理器
WAKMan: Workload Aware KVCache Manager
AI 导读
WAKMan 是一个负载感知的 KV cache 管理器，无需知道请求类型或读取提示词，即可根据实时流量选择淘汰策略，平均将命中率较 LRU 提升 5 个百分点，并在所有策略中取得最低的平均与最差情况最优性差距。
正文 · 原文
Abstract:Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory resource, the KV cache eviction policy becomes a decisive factor in determining the overall benefits of prefix caching. Production traffic may have single-turn, multi-turn, agentic, or a mixture of any combination of these request types. Our findings indicate that agentic steps return within seconds and favor recency, while popular prompts and long conversations return after many other requests and favor frequency. Across 20 mixes, no single policy is always best: vLLM's default strategy LRU provides 12% higher cache hits as compared to the frequency-oriented S3FIFO on agentic-heavy mixes but trails it by up to 22% elsewhere, and even a single workload parameter can flip the winner even for any particular mix. We present WAKMan, a workload-aware KV cache manager that picks its eviction policy from live traffic without knowing request types or reading prompts. It improves hit rate over LRU by 5 percentage points on average, with the lowest mean and worst-case optimality gap of all policies.
| Comments: | Work done during internship at Capital One |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.36355 [cs.AI] |
| (or arXiv:2610.36355v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36355 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vaibhavi Lokegaonkar [view email]
[v1]
Mon, 26 Oct 2026 22:19:37 UTC (502 KB)
来源：arXiv cs.AI · arxiv.org