arXiv cs.CL· Tomas Rasymas, Povilas Kvedaras·· 12 小时前AI 评分31
英语校准侵蚀倍专家剪枝 MoE 模型中的立陶宛语能力
English Calibration Erodes Lithuanian in Expert-Pruned Mixture-of-Experts Language Models
AI 导读
研究用 REAP 对四个 MoE 模型做专家剪枝，剪掉 25% 或 50% 专家后发现英语校准无法迁移到立陶宛语：25% 剪枝下英语理解中位数仅降 1.7 分，立陶宛语却降 21.4 分。
正文 · 原文
Abstract:Expert pruning compresses Mixture-of-Experts (MoE) language models by removing the experts that a calibration set marks as unimportant. Studies of high-resource languages largely find that English calibration transfers well to other languages. We test this for Lithuanian: we pruned four MoE models with REAP, removing 25% or 50% of the experts with English, Lithuanian or mixed calibration, and evaluated both languages on reading comprehension, translation and language modelling. English calibration does not transfer to Lithuanian: at 25% pruning, English comprehension fell by a median of 1.7 points but Lithuanian by 21.4. The transfer is asymmetric. Lithuanian calibration removed most of the Lithuanian loss and never significantly hurt English comprehension, which was on average 2.5 points better than under English calibration; only English language modeling got slightly worse. Routing suggests why: in the last layers, Lithuanian sends 55-72% of its expert selections to just eight experts, against 19-39% for English, and English calibration removes some of these experts. Compressed models should be calibrated, and evaluated, in every language they must serve.
| Comments: | 41 pages, 13 figures, 10 tables |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.07654 [cs.CL] |
| (or arXiv:2611.07654v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07654 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tomas Rasymas [view email]
[v1]
Thu, 5 Nov 2026 08:39:28 UTC (501 KB)
来源：arXiv cs.CL · arxiv.org