arXiv cs.CL· Baran Atalar, Selly Beyene, Xutong Liu, Carlee Joe-Wong·· 12 小时前AI 评分38
面向低成本 LLM 服务的多轮语义缓存方法 CUCB-SC-MT 与 CLCB-LS-MT
Multi-Turn Semantic Caching for Low-Cost LLM Serving
AI 导读
研究提出 CUCB-SC-MT 与 CLCB-LS-MT 两种多轮语义缓存方法，从评判反馈中学习"复用风险"，按预算选择要缓存的对话状态。在 SocraTeach 数学辅导和 Taskmaster-3 电影票务对话上，两者在各缓存预算下损失均低于 ContextCache、ElastiCache 与 vCache 的多轮变体及单轮语义缓存。
正文 · 原文
Abstract:Semantic caching lowers the cost of serving large language models (LLMs) by reusing stored responses for similar requests. In multi-turn conversations, whether a cached response is acceptable depends on both conversations' histories, which embedding similarity of requests and histories predicts poorly (AUC 0.62--0.74 on five workloads). We therefore learn the \emph{reuse risk}, the probability that a cached response is unacceptable, from judge feedback, and choose a budgeted set of conversation states to cache by weighing the generation cost a reuse avoids against a price for unacceptable reuse. We propose CUCB-SC-MT, which learns a cache from logged conversations with a suboptimality guarantee, and CLCB-LS-MT, which learns online and has $\tilde O(\sqrt N)$ regret over $N$ requests when conversation states recur. On math-tutoring (SocraTeach) and movie-ticketing (Taskmaster-3) conversations, both achieve lower loss than prior semantic caching approaches: ContextCache, multi-turn variants of ElastiCache and vCache, and single-turn semantic caching at every cache budget. With 1,000 cache entries, CLCB-LS-MT avoids 35--41\% of the generation cost through acceptable reuses, against at most 22--25\% for the baselines, and lowers estimated serving latency by 19--35\%.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2611.01077 [cs.LG] |
| (or arXiv:2611.01077v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.01077 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Baran Atalar [view email]
[v1]
Sat, 31 Oct 2026 16:16:16 UTC (797 KB)
来源：arXiv cs.CL · arxiv.org