arXiv cs.CL· Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou·· 10 小时前AI 评分38
NVLinkOneSided：面向 NVLink 域内 MoE 推理的单边专家并行 all-to-all
One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink
AI 导读
研究者提出 NVLinkOneSided，一种面向单 NVLink 域内 GPU 的单边 EP all-to-all 方案，源 rank 无需先交换元数据即可将 token 推入目标 GPU 的 rank-major 缓冲区，分组 GEMM 原地读取。
正文 · 原文
Authors:Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou
Abstract:Expert parallelism (EP) shards the experts of a Mixture-of-Experts (MoE) layer across GPUs, so every MoE layer requires two all-to-all communications, namely dispatch and combine. Compared with training, inference processes relatively few tokens per GPU at a time, and at small batch sizes fixed costs that do not grow with the number of tokens, rather than data movement, dominate the latency. We present NVLinkOneSided, a one-sided EP all-to-all for GPUs within a single NVLink domain. Each source rank pushes its tokens, without first exchanging metadata, into rank-major buffers on the target GPUs, and the grouped GEMMs read them in place instead of from a per-expert copy. We provide two protocols for the synchronization that the communication requires, namely Fence and Sentinel, for high-throughput and low-latency scenarios respectively. On Blackwell GPUs, NVLinkOneSided completes dispatch and combine 1.2-3.0$\times$ faster than DeepEP, with the largest speedups at small batch sizes. The resulting MoE layer is up to 2.4$\times$ faster than DeepEP with DeepGEMM. It even outperforms the fused DeepGEMM MegaMoE at up to 128 tokens per GPU, making it still competitive for decoding. We expose NVLinkOneSided through a modular interface, so that it is agnostic to the activation quantization format, decoupled from the MoE compute backend and easy to integrate into inference frameworks.
| Comments: | 19 pages, 8 figures, 14 tables |
| Subjects: | Distributed, Parallel, and Cluster Computing (cs.DC); Computation and Language (cs.CL); Machine Learning (cs.LG); Performance (cs.PF) |
| ACM classes: | C.1.4; D.1.3 |
| Cite as: | arXiv:2611.13188 [cs.DC] |
| (or arXiv:2611.13188v1 [cs.DC] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13188 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bo Li [view email]
[v1]
Mon, 9 Nov 2026 20:06:38 UTC (109 KB)
来源：arXiv cs.CL · arxiv.org