arXiv cs.CL· Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl·· 13 小时前AI 评分18
MoVoC：面向吉兹文字语言的多形态感知子词构建方法
MoVoC: Morphology-Aware Subword Construction for Geez Script Languages Corrected version
AI 导读
MoVoC 提出一种融合监督式形态分析与 BPE 的混合分词方案，并训练出 MoVoC-Tok 分词器，在四种吉兹文字语言中的三种取得最高 MorphScore，在两种语言中取得最高语素边界精确率。团队同时发布四种语言共 169,806 条语素标注数据及其中两种语言的语素感知词表，在未参与训练的 Tigre 跨语言应用上增益最明显。论文已撤回原版本中的机器翻译结果，数据、分词器与评测代码均已公开。
正文 · 原文
Abstract:Subword-based tokenization methods often fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphologically complex languages such as those written in the Ge'ez script. To address this, we present Morpheme-aware Subword Vocabulary Construction (MoVoC) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary. This hybrid segmentation approach combines morpheme-based and Byte Pair Encoding (BPE) tokens to preserve morphological integrity while maintaining lexical meaning. To tackle resource scarcity, we release morpheme-annotated data for four Ge'ez script languages (169,806 records) and a morpheme-aware vocabulary for two of them. Scoring every comparison on words held out from the morpheme resource that constrains the MoVoC-Tok applied to that language, MoVoC-Tok has the highest MorphScore in three of four languages and the highest morpheme boundary precision in two of four, with its largest and most clearly resolved gains on Tigre, a language it was not trained on and to which it is applied cross-lingually. The machine translation results reported in the published version are withdrawn. Data, tokenizers and evaluation code are publicly available.
| Comments: | Corrected version |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; I.2.6; H.3.3 |
| Cite as: | arXiv:2509.08812 [cs.CL] |
| (or arXiv:2509.08812v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2509.08812 arXiv-issued DOI via DataCite |
Submission history
From: Hailay Kidu Teklehaymanot [view email]
[v1]
Wed, 10 Sep 2025 17:45:10 UTC (310 KB)
[v2]
Wed, 21 Oct 2026 21:21:42 UTC (202 KB)
来源：arXiv cs.CL · arxiv.org