arXiv cs.AI· Fa\"icel Chamroukhi, Nhat Thien Pham·· 3 天前AI 评分22
面向分布式混合专家模型的最优传输聚合方法
Optimal Transport Aggregation for Distributed Mixture-of-Experts
AI 导读
针对分布式数据集上独立训练的本地 MoE 模型难以合并的问题，研究者提出通过最小化期望传输散度，将未对齐专家的大混合模型压缩为 K 专家 MoE。该方法用 majorization-minimization 算法计算专家、再拟合门控网络，仅需单轮单向通信，聚合估计量与本地估计量同样具有一致性。模拟与真实数据实验显示，其精度接近集中式估计，计算时间仅为后者一小部分。
正文 · 原文
Abstract:This paper considers the setting in which local Mixture-of-experts (MoE) models are fitted independently on distributed datasets and must be combined into a single global MoE model. Weighted averaging the local mixtures, whose experts are not aligned across machines, often yields a large mixture with many experts instead of $K$ and does not provide an estimator of the true model parameters. We propose to reduce this large mixture to a $K$-expert MoE by minimizing an expected transportation divergence, in which the transport plan depends on the covariates. A relaxation of this problem separates the experts from the gating network: the experts are computed by a majorization--minimization algorithm, which decreases the objective monotonically, and the gating network is then fitted to the gates implied by the optimal plan, with an explicit bound on the resulting loss. We show that the aggregated estimator is consistent whenever the local estimators are, without requiring the local models to have the same labeling across machines. The procedure only needs a single one-way communication round. Experiments on simulated and real data show accuracy close to centralized estimation at a fraction of its computation time. Our source code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (stat.ML) |
| Cite as: | arXiv:2312.09877 [cs.LG] |
| (or arXiv:2312.09877v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2312.09877 arXiv-issued DOI via DataCite |
Submission history
From: Faicel Chamroukhi [view email]
[v1]
Fri, 15 Dec 2023 15:26:13 UTC (669 KB)
[v2]
Wed, 11 Mar 2026 12:36:20 UTC (655 KB)
[v3]
Mon, 12 Oct 2026 08:31:40 UTC (363 KB)
来源：arXiv cs.AI · arxiv.org