arXiv cs.CL· Chen Hu, Yintao Tai, Antonio Vergari, Frank Keller, Alessandro Suglia·· 2 天前AI 评分26
MIXAR：将自回归像素级语言模型扩展至多语言与多文字系统
MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts
AI 导读
MIXAR 是首个生成式多语言像素级语言模型，在八种高资源语言（覆盖拉丁与 CJK 文字）上预训练，参数规模扩展至 0.5B。它用基于 GRPO 的强化学习阶段替代此前像素模型不稳定的对抗训练，在多语言基准上把自由生成准确率最高提升 2.3 倍，并在判别式基准上与同类 tokenizer 模型持平。
正文 · 原文
Abstract:Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges; however, existing models remain English-centric and are hard to train as generators. This paper introduces MIXAR, the first generative multilingual pixel-based language model, pretrained on eight high-resource languages spanning Latin and CJK scripts and scaled up to 0.5B parameters. Although pretrained on high-resource languages only, MIXAR generalizes to unseen low-resource languages better than a tokenizer-based model pretrained on the same corpus, with its largest gains on the scripts most distant from the pretraining mix, taking a step toward more equitable language technology. To make free generation work in pixel space, MIXAR replaces the notoriously unstable adversarial training stage of prior pixel-based models with a GRPO-based reinforcement-learning stage that rewards the model on its own free-running generations: it proves more stable and multiplies free-generation accuracy by up to 2.3x on multilingual benchmarks. Finally, MIXAR performs on par with comparable tokenizer-based models on discriminative benchmarks, improves consistently with scale, and withstands orthographic attacks more gracefully than tokenizer-based models. Code and models are available at this https URL.
| Comments: | 29 pages. Code and models: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.11575 [cs.CL] |
| (or arXiv:2604.11575v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.11575 arXiv-issued DOI via DataCite |
Submission history
From: Yintao Tai [view email]
[v1]
Mon, 13 Apr 2026 14:53:19 UTC (3,930 KB)
[v2]
Thu, 15 Oct 2026 17:46:36 UTC (3,535 KB)
来源：arXiv cs.CL · arxiv.org