arXiv cs.CL· Dipto Sumit, Mahima Mahzabin Sandria, Farig Sadeque, Md. Golam Rabiul Alam·· 11 小时前AI 评分31
研究揭示：Token 并非难度单位，分词方式决定词内自适应计算分配
A Token Is Not a Unit of Difficulty: Segmentation Controls Where Adaptive Computation Goes Inside a Word
AI 导读
针对 Qwen3 的离线分析发现，基于置信度的早退策略在英语、德语、土耳其语中给单词后续片段分配的计算少于首片段，在孟加拉语中却更多，分词方式是主因。用 BPE-dropout 重新分词可使两个自适应轴上的差距最多移动 8 个出口层，BLOOM 和 mGPT 因对孟加拉语切分更粗而呈现英语模式。
正文 · 原文
Abstract:Early-exit language models decide for each token how much computation to spend, usually by stopping once the prediction is confident. Yet a token is not a fixed unit: tokenizers cut words into different numbers of pieces in different languages. We ask where inside a word such a policy spends its budget. In Qwen3, along two adaptive axes (the timesteps of a spiking conversion and the layers of the original Transformer), confidence stopping gives the continuation pieces of a word less computation than its first piece in English, German, and Turkish, but more in Bangla. Segmentation drives this. Re-segmenting the same parallel passages with BPE-dropout shifts the gap by up to eight exits on both axes, small models trained under two tokenizers shift Bangla's gap by six exits, and BLOOM and mGPT, which split Bangla less finely, give it the English pattern. Characters per token do not predict the gap across languages; within-word surprisal explains 75% of its variance. A learned stopping head beats confidence stopping by 7 to 8 points, but gains less than one point over the best mix of fixed exits at equal cost, and sometimes loses. All results come from offline analysis, not measured speedups.
| Comments: | 17 pages, 12 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.00983 [cs.CL] |
| (or arXiv:2611.00983v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00983 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Farig Sadeque [view email]
[v1]
Sat, 31 Oct 2026 14:23:37 UTC (172 KB)
来源：arXiv cs.CL · arxiv.org