arXiv cs.CL· Joel Dag, Nikit Srivastava, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo·· 3 天前AI 评分24
多语言 MoE 如何随语言覆盖改变容量与算力分配
Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation
AI 导读
研究提出一种考虑重复的联合缩放定律，关联激活参数量、每语言训练 token 数、专家数量与语言覆盖，基于 64 个预训练配置的 26,300 条单语言 loss 观测拟合，在留出配置上预测误差比最强基线降低 28%。
正文 · 原文
Abstract:Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20247 [cs.CL] |
| (or arXiv:2610.20247v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20247 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nikit Srivastava [view email]
[v1]
Wed, 14 Oct 2026 14:20:19 UTC (248 KB)
来源：arXiv cs.CL · arxiv.org