arXiv cs.CL· Swapnil Gandhi, Siva Kumar Sastry Hari, William J. Dally, Christos Kozyrakis·· 12 小时前AI 评分38
PACE：为 LLM 服务中的分支并行调度节流
Pacing Branch Parallelism in LLM Serving
AI 导读
PACE 是一种逐步准入控制器，把 LLM 输出中的额外分支视为机会性宽度，仅当预测的分支外部性落在批次当前 slack 预算内时才准入。在 Qwen3-32B 上，PACE 相比无分支并行服务提升 goodput 1.77 倍、相比 eager 准入提升 1.48 倍，并保持 99% SLO 达成率。在另外五个密集与 MoE 模型（最高 671B 参数）上取得 1.72–2.15 倍的可比增益。
正文 · 原文
Abstract:Recent methods expose intra-request parallelism in LLM outputs, allowing independent branches to decode concurrently. Existing serving systems execute these branches eagerly or under fixed caps. We show that both are brittle: eager admission inflates the shared decode step, degrading co-batched requests in serial stages, while conservative fixed caps forgo the throughput that motivated exposing branches in the first place. We call the excess step latency caused by admitted branches the \emph{branch externality} and show that the safe width depends on batch composition, context lengths, and accumulated slack, all of which change continuously over a workload trace. We introduce PACE, a per-step admission controller that treats extra branches as opportunistic width, admitted only when the predicted branch externality fits within the batch's current slack budget. Per-step regulation is practical because branch-level scheduling decouples compute from memory: branches share the request's prefix KV, so expanding or contracting width requires no memory reclamation. On Qwen3-32B, PACE improves goodput by $1.77\times$ over serving without branch parallelism and by $1.48\times$ over eager admission while maintaining 99\% SLO attainment. Comparable gains ($1.72$--$2.15\times$) hold on five additional dense and mixture-of-experts models with up to 671B parameters. Code is available at this https URL
| Comments: | NeurIPS'26 | Camera-Ready |
| Subjects: | Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.06914 [cs.DC] |
| (or arXiv:2605.06914v2 [cs.DC] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06914 arXiv-issued DOI via DataCite |
Submission history
From: Swapnil Gandhi [view email]
[v1]
Thu, 7 May 2026 20:23:32 UTC (252 KB)
[v2]
Sun, 1 Nov 2026 23:01:11 UTC (288 KB)
来源：arXiv cs.CL · arxiv.org