斯坦福研究：BPB 等指标并非完全与分词器无关
斯坦福：BPB 等指标并非完全与分词器无关
斯坦福论文发现没有任何指标能在所有意义上与分词器无关：未训练模型什么都没学到，BPB 仍随分词器变化，英文文本在约 24K 词表时 BPB 最好，但准确率仍为随机水平。在不同分词器训练的模型上，BPB 与任务分数的相关性仅在 +0.02 到 −0.30 之间。研究者建议比较分词器时不要只看单一分数，还需在同一文本上检查任务准确率。
Rohan Paul· @rohanpaul_ai · X·2026-11-06 04:40· 39 分钟前AI 评分45
AI 导读
斯坦福论文指出，没有任何指标能在所有意义上与分词器无关，因此比较不同分词器时不能只信单一分数，还要检查模型是否答对。研究用未训练模型和同一配方不同分词器的已发布模型测试：什么都没学到的模型 BPB 仍随分词器变化，英文文本在约 24K 词表时 BPB 最好，但准确率仍为随机水平。在不同分词器训练的模型上，BPB 与任务分数的相关性仅在 +0.02 到 −0.30 之间。
正文 · 原文
New Stanford paper finds that no metric can be tokenizer-agnostic in every sense, so anyone comparing tokenizers should not trust a single score and should also check whether the model answers correctly.
To compare models with different tokenizers, researchers usually report bits-per-byte, or BPB, which divides the loss by bytes instead of tokens. Many papers call it tokenizer-agnostic.
They scored untrained models that guess uniformly, plus released models that share one recipe but use different tokenizers. They also rechecked a published claim that another supposedly tokenizer-neutral metric agrees with BPB.
Models that learned nothing still got different BPB by tokenizer, with English text scoring best near a 24K vocabulary, while accuracy stayed at chance. On models trained with different tokenizers, correlations between BPB and task scores ran only from +0.02 to −0.30.
When comparing tokenizers, don't treat lower BPB as proof of a better model, and check task accuracy on the same text before picking a vocabulary.
来源：Rohan Paul · x.com
来源：RadarAI · aihot.news