arXiv cs.AI· Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel·· 9 天前AI 评分28
RaReCache：基于秩不一致选择性重算的跨模型 KV Cache 复用框架
RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation
AI 导读
RaReCache 通过秩不一致度量识别信息密集 token，仅重算部分位置即可让大模型准确复用小模型预填充的 KV Cache。在 Qwen3-0.6B 到 14B 的 23 倍参数差距下，重算 30% 位置保留 95-99% 目标精度；Llama3-8B 到 70B 的 8.8 倍差距下，重算 40% 保留 96.5% 精度，并实现最高 3.04x 预填充加速。
正文 · 原文
Abstract:Cross-model KV-cache reuse remains a key challenge in modern LLM serving. Coding agents and multi-model systems increasingly route a shared context across models: a user may switch models mid-session, or a cascade may escalate a difficult query. Because KV caches contain model-specific representations, each switch typically forces the receiving model to prefill the entire context from scratch. Recent work shows that closed-form linear maps can translate KV caches between models in the same family, but transfer accuracy degrades as the model-size gap widens. In this paper, we establish that these transfer failures are concentrated in a small subset of information-dense tokens. To bridge this gap, we introduce RaReCache, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation. RaReCache identifies these critical positions using a novel rank disagreement metric, scoring each token by the energy of its mapped KV in output directions weakly supported by the calibration data. Across two model families and five benchmarks, on a 23x parameter gap (Qwen3-0.6B to 14B) recomputing just 30% of positions retains 95-99% of the target accuracy, whereas on a 8.8x gap (Llama3-8B to 70B), recomputing 40% retains 96.5% of the target accuracy. RaReCache largely removes sensitivity to source-model size, and achieves up to a 3.04x prefill speedup. For online serving, it handles 1.8x the request throughput of target prefill on a single GPU, and at the target's saturation load, reduces median and 99th-percentile time-to-first-token (TTFT) by 5.0x and 6.4x respectively, with a 30% recompute budget. RaReCache establishes an efficient serving paradigm where small models prefill on behalf of massive targets, enabling large models to recompute only critical tokens, drastically reducing prefill latency.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.11358 [cs.AI] |
| (or arXiv:2610.11358v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11358 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sreetama Sarkar [view email]
[v1]
Thu, 8 Oct 2026 06:49:28 UTC (1,103 KB)
来源：arXiv cs.AI · arxiv.org