arXiv cs.CL· Felicia K\"orner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. Hedderich·· 6 天前AI 评分24
多语言预训练中翻译能力如何发展：先复制，后翻译
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
AI 导读
研究者预训练了一个覆盖九种语言的 1.7B 多语言模型，以词级翻译为测试任务追踪翻译能力的早期发展轨迹。结果显示，模型在快速习得基础语言能力的同时进行 token 级复制，翻译则分两阶段发展：初期由复制和表层相似性主导，后期才形成更具泛化性的翻译机制并细化复制。该工作获 EMNLP Main 2026 接收，代码、模型 checkpoint 和数据集均已公开。
正文 · 原文
Abstract:Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We use word-level translation as a testbed, introducing a novel dataset to trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.
| Comments: | 9 pages, accepted to EMNLP Main 2026, code, model checkpoints, and dataset available |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.17633 [cs.CL] |
| (or arXiv:2604.17633v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.17633 arXiv-issued DOI via DataCite |
Submission history
From: Felicia Körner [view email]
[v1]
Sun, 19 Apr 2026 22:03:29 UTC (2,752 KB)
[v2]
Fri, 26 Jun 2026 09:55:26 UTC (3,231 KB)
[v3]
Thu, 8 Oct 2026 19:57:06 UTC (3,212 KB)
来源：arXiv cs.CL · arxiv.org