arXiv cs.CL· Johin Johny Arimbur, Ankur Ashit Ghosh, Shervin Sam·· 10 小时前AI 评分38
2026 年印度语言 Token 税：九款公开 tokenizer 的 Token 差异与成本测算
The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India
AI 导读
研究测量了 GPT-4、GPT-4o、Qwen2.5、DeepSeek-V3、Mistral-Nemo 及四款 Sarvam 模型共九款 tokenizer 在印度 25 种语言上的 Token 开销差异。
正文 · 原文
Abstract:Commercial language model APIs bill per token, so a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window. We measure this disparity for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200 (2,009 professionally translated sentences aligned with English) and on 13,732 human-translated Hinglish-English social media pairs from PHINC. On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script; GPT-4's tokenizer needs up to 12.8 times. Sarvam-30B/105B is the only third-party tokenizer that stays below 2.6 times on every language (median 1.53). In an illustrative scenario using official list prices accessed on 5 November 2026 and stated assumptions, a support bot handling 100,000 chats a month on GPT-4o would cost about INR 56,640 in English and INR 102,829 in Marathi; these are estimates, not measured production costs. Against an openly licensed Hindi and Marathi morphological segmentation set, no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48). In a companion paper we present a tokenizer that reduces this tax. Code, pinned artifacts and raw token counts are available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13058 [cs.CL] |
| (or arXiv:2611.13058v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13058 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Johin Arimbur [view email]
[v1]
Mon, 9 Nov 2026 17:24:21 UTC (473 KB)
来源：arXiv cs.CL · arxiv.org