arXiv cs.CL· Maitrey Mehta, Nishant Subramani, Zhichao Xu, Ashim Gupta, Vivek Srikumar·· 11 小时前AI 评分32
基于可解释性的词表扩展：FragMend 如何修复 LLM 的 token 过度碎片化
Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion
AI 导读
研究提出基于可解释性的词表扩展方法，挑战了传统按词频挑选新增词表项的做法，并在非拉丁文字语言上比基线嵌入初始化方法提升约 20 分。团队还发现模型各层会逐步将碎片化子词合并为更大子词的“子词去分词”现象，并据此提出 FragMend 方法以推高可解释性扩展的效率上限，成果发表于 COLM 2026。
正文 · 原文
Abstract:All languages are equal; when it comes to tokenization, some are more equal than others. Tokens are the hidden currency that dictates the cost and latency of contemporary LLM access. However, many languages written in non-Latin scripts observe a poor exchange rate: LLMs take several multiples of tokens to encode the same information in many languages as they do for English. We observe that this issue, known as token overfragmentation, persists in modern open-weight LLMs. The standard remedy is vocabulary expansion that adds target language items missing from the model's vocabulary. We focus on two core decisions in the vocabulary expansion process: What items should we add? and How should we initialize their corresponding input and output embeddings? We comprehensively study and advance the new research direction of interpretability-based vocabulary expansion along these two core decisions, making four key contributions: First, we question the conventional use of frequency-based methods to choose candidate vocabulary items to add (a decision long treated as settled), and show that interpretability-based methods offer a superior performance-token efficiency tradeoff. Second, we strengthen the case for interpretability-based embedding initialization by showing large gains (~20 pts) over baseline initialization methods for several languages written in non-Latin scripts. Third, we identify the phenomenon of "subword detokenization," where models progressively merge fragmented subword tokens into larger subwords across layers. Fourth, we propose FragMend, a method that leverages this phenomenon to push the efficiency ceiling of interpretability-based expansion, and validate its effectiveness.
| Comments: | Published at COLM 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.16656 [cs.CL] |
| (or arXiv:2604.16656v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.16656 arXiv-issued DOI via DataCite |
Submission history
From: Maitrey Mehta [view email]
[v1]
Fri, 17 Apr 2026 19:21:11 UTC (945 KB)
[v2]
Sun, 1 Nov 2026 23:31:34 UTC (1,241 KB)
来源：arXiv cs.CL · arxiv.org