LangMAP：面向多语言模型的语言自适应 Tokenization 方法

View PDF HTML (experimental)

Abstract:Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. Using one with an existing pretrained model requires either training a new model from scratch or adapting the vocabulary of the pretrained one. We propose Language-adaptive Maximum a Posteriori (LangMAP) Tokenization, which extends the UnigramLM algorithm to the multilingual setting by replacing its single distribution over the vocabulary with one distribution per language over the same vocabulary. LangMAP can be used to train a multilingual language model from scratch, or to adapt a pretrained model's tokenizer to individual languages without changing its vocabulary. Language labels are required to estimate the per-language distributions; at inference, the algorithm selects the highest-probability segmentation across languages, i.e., it does not need a language label. In experiments where we adapt a pretrained tokenizer with LangMAP, we observe that LangMAP's new segmentation has consistently higher morphological boundary recall than the original tokenizer. In code data, it improves alignment with abstract syntax tree (AST) leaf boundaries in all configurations we try. Despite ostensible improvements in segmentations, in language model fine-tuning experiments, we do not observe significant differences in performance for most language-task pairs.

Comments:	Published in Proceedings of EMNLP 2026 (main conference; oral presentation)
Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2606.23566 [cs.CL]
	(or arXiv:2606.23566v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2606.23566 arXiv-issued DOI via DataCite

Submission history

From: Clara Meister [view email]
[v1] Mon, 22 Jun 2026 16:32:00 UTC (606 KB)
[v2] Tue, 23 Jun 2026 17:41:34 UTC (606 KB)
[v3] Mon, 26 Oct 2026 17:26:50 UTC (222 KB)