arXiv cs.CL· Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott·· 13 小时前AI 评分34
扩展多语言预训练：MPIXEL 像素语言模型研究
Scaling Multilingual Pretraining of Pixel Language Models
AI 导读
研究者训练了一系列 MPIXEL 像素语言模型，沿更多语言、更多文字系统和固定单语言算力下更大模型容量三个维度扩展预训练。在覆盖 83 种语言的五项任务评测中，扩展效应是局部的：各维度只提升新增语言和文字系统的表现，而增大容量可缓解多语言预训练带来的“多语言诅咒”。
正文 · 原文
Abstract:Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.
| Comments: | 33 pages, 25 figures, 14 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| MSC classes: | 68T50 |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2610.29681 [cs.CL] |
| (or arXiv:2610.29681v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29681 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: İlker Kesen [view email]
[v1]
Wed, 21 Oct 2026 12:57:41 UTC (1,248 KB)
来源：arXiv cs.CL · arxiv.org