arXiv cs.AI· Oana Balmau, Saurav Banka, Jiaxuan Chen, Anne-Marie Kermarrec, Ramya Narasimha Prabhu, Rafael Pires, Mathis Randl, Martijn de Vos·· 10 小时前AI 评分31
BubbleTea:利用专家并行气泡进行微调
Harvesting Expert-Parallel Bubbles for Fine-Tuning
AI 导读
BubbleTea 是首个利用专家并行气泡做微调的共置系统,通过预测每层每个 GPU 的气泡并填入复用已加载权重的短微调任务,在四个 A100 上的三个 MoE 模型上微调吞吐比 LLMStation 快 1.9–6.2 倍。在突发生产流量下,其训练速度比 LLMStation 快 3.3 倍、比 CUDA MPS 下的原生训练器快 14 倍,p90 推理延迟保持在单独服务水平的 7% 以内。
正文 · 原文
Abstract:GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time.
We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine.
Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2$\times$ faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3$\times$ faster than LLMStation and 14$\times$ faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.
| Comments: | 12 pages, 8 figures |
| Subjects: | Operating Systems (cs.OS); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2611.24496 [cs.OS] |
| (or arXiv:2611.24496v1 [cs.OS] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24496 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ramya Prabhu [view email]
[v1]
Wed, 18 Nov 2026 14:24:23 UTC (955 KB)
来源:arXiv cs.AI · arxiv.org