arXiv cs.AI· Hongyao Liu, Liuqun Zhai, Junyi Wang, Zhengru Fang, Jingshu Chen, Jun Huang·· 9 天前AI 评分22
SparKV：面向端侧 LLM 推理的感知开销 KV Cache 加载框架
SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
AI 导读
SparKV 是一个结合云端 KV 流式传输与端侧计算的自适应 KV 加载框架，通过建模各 KV 分块成本来决定流式传输还是本地计算，并重叠两条执行路径以降低延迟。它还能在运行时细化离线生成的调度方案，以应对无线连接和边缘资源波动。实验显示，SparKV 将首 token 时间（TTFT）降低 1.3x-5.1x，单请求能耗降低 1.5x-3.3x，对响应质量影响可忽略。
正文 · 原文
Abstract:Efficient inference for on-device Large Language Models (LLMs) remains challenging due to limited hardware resources and the high cost of the prefill stage, which processes the full input context to construct Key-Value (KV) caches. We present SparKV, an adaptive KV loading framework that combines cloud-based KV streaming with on-device computation. SparKV models the cost of individual KV chunks and decides whether each chunk should be streamed or computed locally, while overlapping the two execution paths to reduce latency. To handle fluctuations in wireless connectivity and edge resource availability, SparKV further refines offline-generated schedules at runtime to rebalance communication and computation costs. Experiments across diverse datasets, LLMs, and edge devices show that SparKV reduces Time-to-First-Token by 1.3$x-5.1x with negligible impact on response quality, while lowering per-request energy consumption by 1.5x to 3.3x, demonstrating its robustness and practicality for real-world on-device deployment.
| Comments: | IoTJ |
| Subjects: | Networking and Internet Architecture (cs.NI); Artificial Intelligence (cs.AI); Performance (cs.PF) |
| Cite as: | arXiv:2604.21231 [cs.NI] |
| (or arXiv:2604.21231v3 [cs.NI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.21231 arXiv-issued DOI via DataCite |
Submission history
From: Hongyao Liu [view email]
[v1]
Thu, 23 Apr 2026 02:55:31 UTC (8,924 KB)
[v2]
Mon, 4 May 2026 22:59:12 UTC (1 KB) (withdrawn)
[v3]
Thu, 8 Oct 2026 06:24:54 UTC (8,743 KB)
来源：arXiv cs.AI · arxiv.org