arXiv cs.CL· Pietro Di Nicco, Lucio La Cava, Andrea Tagarelli·· 12 小时前AI 评分22
AdaBPE：面向混合自然语言与结构化符号语料的自适应频率阈值 BPE 分词方法
Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora
AI 导读
研究者提出 AdaBPE 分词方法，通过自适应频繁模式挖掘，按候选 token 长度和类型动态调整最小频率阈值，避免偏向高频短模式，并区分通用语言与 LaTeX 等形式语言模式。在覆盖 LaTeX 与自然语言的五个数据集上，AdaBPE 在领域特定内容的编码效率和下游语言模型性能上均取得一致提升，且不损害通用语言表现。
正文 · 原文
Abstract:Modern language models rely on subwording tokenization to define the discrete units of input processing, typically using methods like Byte Pair Encoding (BPE) that treat whitespace as a fixed semantic boundary. This design is inherently limited, as many meaningful concepts---including multiword expressions and the structured environments of formal languages like LaTeX---frequently span across whitespaces. While recent superword tokenization methods attempt to address this by relaxing whitespace constraints, they often rely on greedy frequency-driven merging that fails to account for token length or the specific needs of underrepresented domain-specific languages. In this paper, we introduce Adaptive Frequency Threshold Byte Pair Encoding (AdaBPE), a new tokenization method that leverages an adaptive frequent-pattern mining approach. AdaBPE features a twofold dynamicity: it adapts minimum frequency thresholds based on both the length of a candidate token, to prevent favoring frequent short patterns, and its type, to distinguish between general and formal language patterns, using LaTeX as case in point. This allows the model to efficiently allocate vocabulary capacity to domain-specific notations, such as LaTeX, even when they represent a small fraction of the training data. We evaluate AdaBPE across five datasets spanning LaTeX and natural language, showing consistent gains in encoding efficiency and downstream language-model performance on domain-specific content without compromising general-language performance.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.32921 [cs.CL] |
| (or arXiv:2610.32921v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32921 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Andrea Tagarelli [view email]
[v1]
Fri, 23 Oct 2026 17:21:18 UTC (196 KB)
来源：arXiv cs.CL · arxiv.org