arXiv cs.CL· Warit Sirichotedumrong, Adisai Na-Thalang, Chanakan Wittayasakpan, Kritsadha Phatcharoen, Kunat Pipatanakul·· 10 小时前AI 评分18
khammoon：从语言学准则到泰语语素分词器
khammoon: From a Linguistic Guideline to a Thai Morpheme Segmenter
AI 导读
研究团队提出以泰语最小语义单位 khammoon 进行语素级分词，并据此制定形式化决策准则（人工一致率 κ≈0.99），用 LLM 扩展标注、人工修正系统性错误。基于该数据训练的 khammoon 是 1M 参数的卷积 BiLSTM 分词器，面向 CPU 高效推理，域外文本边界 F1 达 0.992。语素级模型在词性标注、命名实体识别和情感分类上持平或超过词级基线，准则、数据集、模型与库均已公开。
正文 · 原文
Abstract:Thai is written without spaces between words, making segmentation a foundational step in most Thai NLP pipelines. Existing tools primarily reproduce the word-level conventions of corpora such as BEST; however, because the notion of a "word" in Thai lacks a unified linguistic consensus, these conventions struggle with productive compounding, leading to arbitrary boundaries and ambiguity in downstream representations. Instead, we propose operating at the morphological level, focusing on khammoon, the smallest atomic unit that carries meaning. Segmenting at the morphemic level provides an explicit, consistent morphological representation that eliminates compounding ambiguity. Crucially, this representation benefits downstream learning: models trained on morphemes perform on par with or exceed word-level baselines across part-of-speech tagging, named-entity recognition, and sentiment classification, demonstrating that linguistic consistency improves representation quality without downstream performance penalty. To operationalize this representation, we translate linguistic principles into a formal decision guideline with near-perfect human agreement ($\kappa \approx 0.99$), scale annotation with an LLM, and correct systematic errors with human labels. Finally, we train khammoon, a lightweight 1M-parameter convolutional BiLSTM segmenter designed for efficient CPU inference, achieving a boundary F1 of 0.992 on out-of-domain text. The guideline, datasets, model, and library are all publicly released.
| Comments: | Technical Report. Code, models, and data are available at this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.13012 [cs.CL] |
| (or arXiv:2611.13012v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13012 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Warit Sirichotedumrong [view email]
[v1]
Mon, 9 Nov 2026 03:19:53 UTC (3,539 KB)
来源：arXiv cs.CL · arxiv.org