arXiv cs.CL· Ben Gubler·· 10 天前AI 评分24
Tokka-Bench：跨 100 种自然语言与 20 种编程语言的 Tokenizer 评测基准
Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages
AI 导读
开源框架 Tokka-Bench 从 bytes per token、unique token coverage、subword fertility、word-split rate 和 vocabulary composition 五个指标，对 100 种自然语言（30+ 种文字系统）和 20 种编程语言的 tokenizer 进行评测。
正文 · 原文
Abstract:Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comparative evaluation. We introduce Tokka-Bench, an open-source framework that evaluates tokenizers on five complementary metrics -- bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary composition -- across 100 natural languages (30+ scripts) and 20 programming languages, using language-aware segmentation adapted to each writing system. Comparing seven BPE tokenizers (GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2) within individual languages, we find that vocabulary allocation strategy matters more than raw vocabulary size, and that programming-language efficiency has converged among recent tokenizers despite divergent natural-language profiles. The framework, data, and interactive dashboard are publicly available.
| Comments: | 5 pages, 5 figures. Code and data: this https URL. Interactive dashboard: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.08794 [cs.CL] |
| (or arXiv:2610.08794v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08794 arXiv-issued DOI via DataCite |
Submission history
From: Ben Gubler [view email]
[v1]
Wed, 25 Mar 2026 21:01:32 UTC (47 KB)
来源：arXiv cs.CL · arxiv.org