arXiv cs.CL· Eugene Jang, Catherine Arnett, Terra Blevins·· 10 小时前AI 评分34
文本编码何时影响语言建模？UTF-8 与多语言编码策略对比研究
When does text encoding matter for language modeling?
AI 导读
研究对比 UTF-8 与其他 Unicode 编码、语言专用编码及自定义编码对字节级语言模型性能的影响，并提出混合编码方案 Aether：对目标语言使用文字系统专用编码，同时以 UTF-8 回退保持完整 Unicode 覆盖。
正文 · 原文
Abstract:UTF-8's role as the de facto encoding standard in language models introduces cross-lingual inequities due to its variable-length design. However, the effect that this encoding choice has on model performance remains largely unexamined. In this work, we investigate how text encoding impacts the performance of byte-level language models, comparing UTF-8 with a wide range of other encoding strategies, including other Unicode-based approaches, language-specific encodings, and our own customized encodings. To facilitate our analysis, we propose Aether: a hybrid encoding scheme that uses script-specific encodings for our target setting while maintaining full Unicode coverage through UTF-8 fallbacks, using this framework to model four orthographically diverse languages. Surprisingly, we find that while languages with information-dense characters (Chinese, Korean) significantly benefit from compressive encodings, languages whose characters carry less information individually (Georgian, Tamil) are largely insensitive to encoding choice. In light of these findings, we argue that text encoding should be a language-specific design decision, rather than a universal default.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.14459 [cs.CL] |
| (or arXiv:2611.14459v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14459 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Eugene Jang [view email]
[v1]
Tue, 10 Nov 2026 18:27:26 UTC (435 KB)
来源：arXiv cs.CL · arxiv.org