arXiv cs.AI· Seongtae Hong, Youngjoon Jang, Seungyoon Lee, Jungseob Lee, Heuiseok Lim·· 3 天前AI 评分24
MixLID：混合语言文本中的 token 级语言识别模型
MixLID: Token-Level Language Identification in Mixed-Language Text
AI 导读
MixLID 是一个支持 111 种语言-文字组合的 token 级语言识别模型，采用两阶段训练：先用语料中的单语标签学习区分各语言，再在混合语言数据上继续训练，第二阶段数据由现有语码转换样本与基于英文源词和借词对应关系做词替换生成的增强样本拼接而成。
正文 · 原文
Abstract:Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 language-script combinations. MixLID follows a two-stage training process: it first learns to distinguish individual languages using the language labels of monolingual examples, then continues training on mixed-language data. The second stage uses concatenated examples from existing code-switching data alongside augmented examples created through word substitution based on correspondences between English source words and loanwords. For monolingual identification, MixLID outperforms all baselines in both accuracy and macro F1 on four benchmarks within the language coverage shared with each baseline. For mixed-language identification, it outperforms the baselines in identifying the language of individual tokens on every evaluated language pair. MixLID achieves high performance across a broad range of languages in both monolingual and mixed-language settings. The model is publicly available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16601 [cs.CL] |
| (or arXiv:2610.16601v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16601 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Seongtae Hong [view email]
[v1]
Mon, 12 Oct 2026 11:21:09 UTC (64 KB)
来源：arXiv cs.AI · arxiv.org