arXiv cs.CL· Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava, Kevin Jiang, Mridul Sharma, Catherine Arnett·· 2 天前AI 评分22
研究：没有单一 tokenizer 特征能可靠预测下游语言模型性能
No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
AI 导读
一项分析约 200 个语言模型（覆盖九种语言）的研究发现，现有各类 tokenizer 内在评测指标均无法在模型家族、任务和语言之间一致地预测下游性能。作者据此提出一个基于语言学的新指标，显示新指标能捕捉现有指标遗漏的 tokenizer 质量维度，但其预测能力仍可能因模型家族、语言和任务而异。该论文已被 EMNLP 2026 Findings 接收。
正文 · 原文
Abstract:Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evidence about which tokenizer features predict a better language model without requiring a full training run. In this paper, we analyze approximately 200 language models trained on nine different languages. We evaluate both models and their tokenizers and test whether there is a relationship between any proposed intrinsic tokenizer evaluations and model performance, as measured by a wide variety of benchmarks. Despite the quantity and variety of proposed intrinsic tokenizer metrics, we find that none consistently predicts downstream performance across model families, tasks, and languages. This motivates the need for novel tokenizer evaluation metrics and approaches. We propose novel linguistically grounded metric introduced in this work shows that new metrics can capture aspects of tokenizer quality that existing metrics miss, though their predictive power is likely to vary across model families, languages, and tasks.
| Comments: | 10 pages, accepted to Findings of EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.21679 [cs.CL] |
| (or arXiv:2610.21679v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21679 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Catherine Arnett [view email]
[v1]
Thu, 15 Oct 2026 11:05:07 UTC (1,035 KB)
来源：arXiv cs.CL · arxiv.org