arXiv cs.AI· Ruicheng Ao, Jing Dong, Gan Luo, David Simchi-Levi·· 13 小时前AI 评分34
内存受限 LLM 推理中的服务自致拥塞机制
Service-Induced Congestion in Memory-Constrained LLM Serving
AI 导读
研究提出内存受限 LLM 推理的离散时间动力学模型，刻画连续批处理下的请求准入、KV cache 增长与驱逐过程。在饱和输入下，无驱逐的固定点不稳定，系统会收敛到唯一的最坏情况极限环，吞吐损失最高达 50%。异构负载中互质解码长度可稳定无驱逐平衡，非互质长度则产生同步模式引发不稳定。
正文 · 原文
Abstract:In large language model (LLM) serving, each request accumulates persistent graphics processing unit (GPU) memory during service as its key-value cache grows with every generated token. Under high concurrency, aggregate memory usage therefore increases endogenously over time: the service process itself creates future capacity pressure. When memory capacity is exceeded, systems evict active requests, discarding cached state and restarting them later, which wastes computation and reduces throughput. We develop a discrete-time dynamical model of memory-constrained LLM inference that captures admission, memory growth, and eviction under continuous batching. In the saturated-input regime, the system admits both eviction-free fixed points and limit cycles with evictions. For homogeneous workloads, we show that the eviction-free equilibrium is unstable and that, except for a Lebesgue-measure-zero exact-capture set, the system converges to a unique worst-case limit cycle that is asymptotically stable outside this exceptional set, with throughput losses as large as 50%. For heterogeneous workloads, we prove a stability criterion in the two-class common-input setting and explain how the survival-polynomial mechanism generalizes to multiple classes and heterogeneous-input lengths. Under an input-dominated scaling regime, coprime decoding lengths stabilize the eviction-free equilibrium, while non-coprime lengths create synchronized modes that drive instability. These results characterize when workload heterogeneity desynchronizes completions and helps stabilize memory-constrained serving. More broadly, we identify service-induced congestion as a structural instability mechanism and derive scheduling design principles for sustaining high throughput.
| Comments: | 100 pages. v2: adds GPU experiments with benchmark workloads (Appendix D.4) and revises the figures |
| Subjects: | Optimization and Control (math.OC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2606.15555 [math.OC] |
| (or arXiv:2606.15555v3 [math.OC] for this version) | |
| https://doi.org/10.48550/arXiv.2606.15555 arXiv-issued DOI via DataCite |
Submission history
From: Ruicheng Ao [view email]
[v1]
Sun, 14 Jun 2026 02:49:03 UTC (3,058 KB)
[v2]
Mon, 19 Oct 2026 03:59:06 UTC (1,354 KB)
[v3]
Tue, 20 Oct 2026 03:22:51 UTC (1,355 KB)
来源：arXiv cs.AI · arxiv.org