arXiv cs.CL· Binayak Mukherjeea, Moitreya Nandi, Arindam Panja·· 12 小时前AI 评分26
AdaptTok-Indic：面向印度语言的字形感知自适应分词
AdaptTok-Indic: Script-Aware Adaptive Tokenization for Indic Languages
AI 导读
AdaptTok-Indic 是自适应分词方法的印度语言专用扩展，结合 Unicode 归一化预处理、显式印度语连字处理、字形感知路由与字节级边界预测，并与小型 Transformer 语言模型联合训练。
正文
Abstract:Multilingual tokenizers trained on English-dominated corpora often over-segment Indic languages, resulting in longer token sequences and increased tokenization disparity across scripts. We present AdaptTok-Indic, an Indic-specialized extension of adaptive tokenization that combines Unicode-normalized preprocessing with explicit Indic conjunct handling, script-aware routing, and learned byte-level boundary prediction jointly trained with a small Transformer language model. Evaluated on Bengali and Hindi, with additional evaluation on Gujarati, Tamil, and Telugu, AdaptTok-Indic achieves lower token fertility than SentencePiece BPE, an Indic-oriented tokenizer baseline, and an independently implemented MAGNET-style baseline. Across five recorded seeds, mean fertility is 1.432 for Bengali and 1.388 for Hindi, compared with 2.156 and 2.085 for SentencePiece BPE. Recorded downstream scores average 57.69 on evaluated Indic settings versus 50.82 for the SentencePiece BPE configuration. Independent verification runs using the released prototype confirm the same directional trends. The current implementation is a small-scale prototype with documented simplifications, and no statistical significance testing, released-MAGNET comparison, or code-mixed evaluation is performed. These results suggest that Indic-specialized adaptive segmentation can reduce tokenization disparity while maintaining competitive downstream performance.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.33017 [cs.CL] |
| (or arXiv:2610.33017v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33017 arXiv-issued DOI via DataCite |
Submission history
From: Arindam Panja [view email]
[v1]
Thu, 8 Oct 2026 14:02:13 UTC (390 KB)
来源：arXiv cs.CL · arxiv.org