Nature 论文提出 byteification：将子词级 LLM 改造为字节级模型

主流

近年来 AI 的进展由端到端深度学习系统推动，这些系统直接从数据中学习表示。大型语言模型（LLM）正是这一趋势的典型代表，它们通过在海量文本集合上训练而获得了强大的能力^3,4。然而，尽管表面上具有通用性，当代 LLM 并非完全端到端：在学习开始之前，文本必须先被映射为一系列称为 token 的离散单元。token 的选择虽然有时被忽视，却从根本上塑造了 LLM 所学到的表示以及它们所表现出的行为^5,6,7,8,9。

绝大多数当代 LLM 使用单词或单词的一部分作为 token，这一过程被称为子词分词^1,10。这带来了许多问题。使用子词分词的 LLM 在字符级理解上存在局限^11,12,13，这尤其会妨碍其在科学数据（如代码和生物序列）上的表现^{2,14,15,16,17}；它们还会因提示如何被分词而隐含地偏向于生成特定回复^18,19,20；它们能纳入词汇表的单词数量受到限制，这在实践中导致以英语为中心^6,21；并且它们可能无法最优地分配其算力^2,22。这些问题促使人们广泛研究子词分词的替代方案，最常见的是使用文本所编码的底层 UTF-8 字节²³作为离散单元。许多早期的字节级 LLM 声称在效率—性能帕累托前沿上优于子词级 LLM^{2,22,24,25,26,27}。然而在实践中，字节级 LLM 迄今尚未得到广泛采用，所有领先的 LLM 仍然完全依赖子词分词。

我们假设，这种理论与实践之间不匹配的关键原因在于，现有的字节级语言建模方法主要侧重于从随机初始化开始训练一个新的字节级模型，并将其与同样从随机初始化开始训练的子词级 LLM 进行比较。相比之下，最先进的子词级 LLM 的训练正在迅速演进，融合了训练数据整理、模型架构和后训练方面的创新。若没有大量投入，字节级 LLM 的开发要跟上这一步伐是不可行的。

为了解决这一不匹配问题，我们提出了一种通用方法，用于创建字节级 LLM，使其在各种任务上的性能接近最先进的子词级 LLM，同时远超参数数量相当的早期字节级 LLM。与主要侧重于从随机初始化开始训练的早期字节级 LLM 不同，我们使用不到典型预训练预算的 1%（总计 491 亿个 token，其中 B 代表十亿或 10⁹）将现有的子词级 LLM 改造为字节级。我们将这一过程称为 byteification。Byteification 在现有子词级 LLM 与字节级 LLM 之间建立了联系。这使我们能够从现有的完全开放 LLM Olmo 3 7B²⁸ 和 OLMo 2 1B²⁹ 出发，训练出 byteified 模型 Bolmo 7B 和 Bolmo 1B；通过 byteify Qwen3 8B Base³⁰ 得到 Bwen 8B；通过 byteify Llama 3 8B³¹ 得到 Blama 8B。

我们的架构首先将字节流聚合成一系列 patch，其中每个 patch 本身由一个或多个字节组成。然后，这些 patch 由一个大型 transformer³² 语言模型处理，最后被解池化为字节。由于其将 tokenization 隐式地转化为字节 patch，我们将这种架构风格称为隐式 tokenizer 语言模型（LTLM）。LTLM 的例子包括 DTP²⁴、BLT²² 和 H-Net² 模型。然而，与早期工作不同，我们专门设计了我们的架构，使其非常适合 byteification（见“Byteified 语言模型架构”一节）。特别是，我们解决了子词 tokenization 的表达能力与隐式 LTLM tokenization 之间的不匹配问题。结合高效的两阶段训练流程（见“Byteification 训练流程”一节），这使我们能够快速恢复，并在某些情况下超越源子词级 LLM 的性能。我们相信，byteification 提供了一个关键缺失的研究方向，使得无需大量投入即可创建最先进的字节级 LLM。这种方法与从随机初始化开始训练是互补的。使任何子词模型都能以低成本进行 byteify，可以快速揭示出有前景的高性能架构，这些架构是作为字节级 LLM 从随机初始化开始训练的有力候选。

平均而言，我们的 byteified 模型在所有公开可用的、规模相当的早期字节级 LLM 中表现更优。例如，Bolmo 7B 在 STEM 任务上比从随机初始化训练的 BLT 7B 取得了 +16.5% 的绝对提升。Bolmo 7B 在字符理解方面也大大优于源 Olmo 3，并在某些编码设置中表现更好。Bwen 8B 优于 Bolmo 7B，展现出类似的趋势，性能接近并有时超过源 Qwen 模型。此外，我们发现 byteified 模型可以通过以更高的每 patch 字节比例进行训练来任意进一步加速，而这在子词级 LLM 中只能在有限程度上实现。最后，我们表明，源 LLM 生态系统中的现有组件可用于以零额外训练成本适配 byteified 模型；这可以进一步加速字节级 LLM 的研究。

总体而言，我们的结果表明，字节级 LLM 作为未来语言模型的基础具有巨大前景，其潜在优势包括：通过降低能源和部署成本来提高计算效率，减少以英语为中心的子词分词引入的偏差，以及支持需要细粒度文本理解的应用，尤其是在科学和技术领域。

字节化语言模型架构

与早期 LTLM 的整体结构相同（方法中的“相关工作”），字节化模型可以如图 1 所示形式化。简而言之，一个在字节上运行的局部编码器模型首先创建上下文化的字节表示。然后，一个神经边界预测器模块基于这些表示，预测在哪些字节位置放置边界。边界信息被用来推导出一系列补丁：一个补丁是连续边界之间的连续字节序列（包括放置结束边界的那个字节）。接着，一个池化模块将上下文化的字节表示缩减为每个补丁的单一表示，该表示通过一个深度全局模型。通过全局模型上下文化的补丁表示被反池化回字节级别，并通过一个局部解码器模型进一步上下文化。最后，局部解码器表示被用来预测下一个字节以及任何其他潜在信息。我们选择局部模型为浅而宽，因为我们发现原始模型的嵌入通常具有较高的有效秩（扩展数据图 1），并且我们选择矩阵长短期记忆（mLSTM）层³³来上下文化局部信息，因为它们在实验中在高吞吐量下仍保持高性能（扩展数据图 3）。有关架构的详细描述，请参见方法中的“字节化模型架构细节”。我们与早期架构的主要不同之处在于非因果的补丁边界预测。早期 LTLM 的边界预测器仅使用过去的上下文来决定是否放置边界（它是因果的，被 Pagnoni 等人²²称为增量式）。乍一看，这似乎是必要的：我们的目标是预测下一个字节，因此我们绝不能泄露任何关于它的信息。然而，尽管子词级 LLM 对子词标记使用因果约束，但子词标记本身并不完全依赖于过去的上下文：子词分词器使用关于未来字节的信息来放置标记边界。为了看清这一点，让我们将子词分词器解释为一个决定是否在任何字节之后放置标记边界的函数：\({\mathcal{B}}(x):{\{0,1,\ldots ,255\}}^{n}\to {\{0,1\}}^{n}\)，其中第 n 位的 1 表示在第 n 个字节之后预测到一个标记边界。让我们假设（1）一个包含英语单词和子词的词汇表，（2）示例文本‘_Hello_Wor!’，它通常会被分词为 {‘_Hello’, ‘_Wor’, ‘!’}，以及（3）位置 i = ∣‘_Hello_Wor’∣ − 1 = 9。\({\mathcal{B}}\) (‘_Hello_Wor!’)_i = 1，因为在‘r’之后有一个边界。然而，在文本‘_Hello_World!’中，它会被分词为 {‘_Hello’, ‘_World’, ‘!’}，我们有 \({\mathcal{B}}\) (‘_Hello_World!’)_i = 0，尽管‘_Hello_Wor!’[:i + 1] = ‘_Hello_World!’[:i + 1] = ‘_Hello_Wor’。换句话说，尽管子词级 LLM 仅使用过去的子词标记来预测下一个子词标记，但子词标记本身是在考虑未来上下文的情况下创建的。在这种情况下，这意味着决定‘_Wor’在一种情况下应该是一个标记，而在另一种情况下则不是，尽管到该点为止的文本是等价的。相比之下，当前的 LTLM 无法考虑未来上下文。这在 LTLM 边界预测器和子词分词器的表达能力之间造成了不匹配。我们修改了边界预测器以解决这种不匹配。特别是，尽管早期的边界预测器可以写成

$${\mathcal{B}}{(\hat{e})}_{t}:= \,f({\hat{e}}_{0},{\hat{e}}_{1},\ldots ,{\hat{e}}_{t}),$$

其中 f 是一个函数，利用截至当前字节表示 \({\hat{e}}_{t}\) 的上下文（其中 t 是当前字节的索引），将字节表示 \(\hat{e}\) 映射为边界分数，我们将边界预测器设为

$${{\mathcal{B}}}_{\mathrm{byteify}}{(\hat{e})}_{t}:= \,f({\hat{e}}_{0},{\hat{e}}_{1},\ldots ,{\hat{e}}_{t},{\hat{e}}_{t+1}).$$

也就是说，我们最多使用 1 字节的未来上下文。具体而言，我们将边界预测器参数化为

$${{\mathcal{B}}}_{\mathrm{byteify}}{(\hat{e})}_{t}:= \,\frac{1}{2}\left(1-\frac{{({W}_{q}{\hat{e}}_{t+1})}^{{\rm{T}}}({W}_{k}{\hat{e}}_{t})}{\parallel {W}_{q}{\hat{e}}_{t+1}\parallel \parallel {W}_{k}{\hat{e}}_{t}\parallel }\right)\in [0,1],$$

其中 W^q 和 W^k 是如 Hwang 等人² 所述的学习投影矩阵，不过我们计算的是当前字节表示与未来一个位置字节表示的投影之间的余弦距离。尽管子词分词器原则上可以不受限制地访问未来信息，但在实践中，我们发现 1 字节的前瞻就足以在很大程度上匹配子词分词的行为。如图 1 所示，考虑未来上下文还能使片段在语义上更加连贯。例如，在包含复合词的文本中，如 ‘the flowerbed’，边界预测器有三个直观选项：(1) 将整个复合词作为一个片段 ‘flowerbed’，(2) 在 ‘r’ 之后放置片段边界以形成片段 ‘flower’，或 (3) 在 ‘b’ 之后放置片段边界（一旦明显这是一个复合词）以形成 ‘flowerb’。选项 (2) 可以说是语义上最连贯的³⁴；然而，对于因果边界预测器而言，这意味着对于每个以 ‘the flower’ 开头的文本，包括例如 ‘the flowers’，都必须在 ‘r’ 之后放置片段边界，而非因果预测器则可以根据未来上下文进行调整。Hwang 等人² 的字节化策略基于选项 (3) 施加监督，即预测下一个子词 token 的开始位置而非前一个的结束位置，这会形成片段 ‘flowerb’， 如图 1 所示。

虽然使用未来上下文对于预填充来说没问题，但我们需要在不观察下一个字节的情况下判断是否放置边界以进行解码。因此，我们在词表中添加了一个特殊符号 <b>，并让本地解码器学习在每个补丁的末尾发出 <b>（相比之下，本地编码器从不会看到 <b>）。这类似于 Fleshman 和 Van Durme³⁵ 中的输出边界预测。实际上，我们最终得到了两个边界预测器：边界预测器 \({\mathcal{B}}\) 接收来自本地编码器的浅层上下文化表示并带有未来上下文（用于预填充期间），以及作为语言建模头一部分的边界预测器，接收来自本地解码器的深层上下文化表示且不带未来上下文（用于解码期间）。值得注意的是，这恰好等同于子词级 LLM 的行为：预填充使用外部子词分词器进行分词（类似于边界预测器 \({\mathcal{B}}\)），而输出边界 <b> 在解码时与每个子词 token 的文本内容一起被隐式预测（类似于我们的输出边界预测器），如图 1 所示。这使得我们的字节化模型既能实现准确的边界，又能获得忠实于原始模型的表示，而早期方法只能二选一（扩展数据表 6）。由于我们的目标是在每个补丁之后预测符号 <b>，我们的本地解码器变成了一个 \({{\mathbb{R}}}^{n\times d}\to {{\mathbb{R}}}^{(n+k)\times d}\) 上的模型，也就是说，本地解码器需要多处理 k 个位置。虽然这种开销原则上并非不可承受，但它使得使用输出边界预测的模型与不使用输出边界预测的模型难以比较。因此，我们通过为每个字节添加一个后跟边界的相同字节版本，将字节词表大小从 256 翻倍到 512，使其实际上零成本。诸如 top_p³⁶ 之类的采样算法随后在大小为 512 的组合词表上运行。那么，本地解码器的目标就是在每一步预测当前字节以及其后是否跟有边界。融合边界符号使本地解码器重新变为各向同性模型。唯一剩余的开销来自对 512 个而非 256 个输出 token 的 softmax，这可以忽略不计。

字节化使总参数量与源子词级 LLM 的参数量保持接近，因为输出嵌入矩阵被移除，但添加了来自本地模型的新参数（方法中的“字节化模型架构细节”和扩展数据表 5）。

字节化训练过程

我们的字节化流程包含两个阶段。在第一阶段，我们旨在快速学习局部编码器、局部解码器、边界预测器和语言建模头的权重。为此，我们引入了一个新目标，以优化并精确恢复源子词级 LLM 的行为（方法中的“字节化流程”）。在此阶段，全局模型的参数保持冻结。在第二阶段，我们训练整个模型，使其学会使用字节级信息，同时可选地提高每个补丁的字节目标压缩比。我们在所有实验中采用了一致的实验设置（方法中的“实验设置”）；完整的训练超参数值列表见扩展数据表 4。

字节化后的性能

我们对 Qwen3 8B³⁰、Llama 3 8B³¹ 和 Olmo 3 7B²⁸ 模型进行了字节化，并将其字节化变体分别称为 Bwen 8B、Blama 8B 和 Bolmo 7B。表 1 将这些模型与源子词级模型以及在 Bolmo 训练设置下使用我们的数据混合继续训练的 Olmo 3 进行了比较。我们还与现有规模相当的字节级 LLM 进行了比较：EvaByte 6.5B²⁷、TFree-HAT 7B³⁷ 和 BLT 7B²²。Bwen 8B 在统计上显著优于所有早期的字节级模型，并且在除 GenQA 之外的每个类别中都有数值上的提升（在此类别中，它略微落后于 TFree-HAT 7B）。Bwen 8B、Blama 8B 和 Bolmo 7B 也接近匹配各自的源模型。Bwen 8B 因其更强大的骨干网络而优于 Bolmo 7B。我们在 1B 评估套件（扩展数据表 2）中观察到了字节化 1B 模型（扩展数据表 3）的类似趋势。

对于 Code 类别，字节化模型在较低的 pass@1 下实现了更高的 pass@16 率，这表明在给定的采样设置下，它们比源子词级模型生成了更多样化的续写，而所有模型的采样设置是相同的（temperature = 0.6 和 top_p = 0.6；扩展数据表 1）。然而，要得出字节级模型实际上更适合生成更多样化生成的结论，需要绘制质量–多样性帕累托前沿，因为观察到的多样性增加也可能由采样设置或引擎对字节级模型与原始子词级模型的不同影响来解释。在未来的工作中探索这一方面可能会很有用（补充信息中的“未来方向”）。

在字符理解方面，字节化模型大幅超越了其子词级对应模型。相比之下，早期的字节级模型并未优于子词级模型。这可以解释为字符理解主要是通过规模获得的¹²，因此尽管字节级模型获取字符理解所需的规模应更小，但 Olmo 3 增加的规模——可能在其他模型多得多的 token 上训练——或许弥补了这一点。Bolmo 7B、Bwen 8B 和 Blama 8B 使用鼓励字符理解的合成数据训练（Methods 中的“Character-level training data”），这加速了该技能的获取。在我们的数据混合上继续训练的 Olmo 3 模型作为对照，它在相同数据上训练，但架构不同。比较 Olmo 3 7B (CT) 和 Bolmo 7B，我们发现在此设置下，该架构导致 MC、GenQA 和 Math 类别略有下降，同时导致 Code (pass@1) 大幅下降，Code (pass@16) 以及字符级理解则显著提升。

提高模型效率

LTLM 的一个显著优势是——与子词级 LLM 不同——它们不局限于固定、有限的 patch 集合。我们接下来研究了能否利用 patch 策略选择上更大的自由度，通过鼓励每个 patch 平均包含更多字节来训练更快的模型。具体而言，我们尝试了将外部边界监督从原始子词分词边界 \({{\mathcal{B}}}_{\mathrm{subword}}\) 改为基于字节对编码（BPE）¹、下一 token 熵或交叉熵选择的这些边界的子集（Methods 中的“Byteifying procedure”）。

作为基线，我们通过将分词器迁移到 SuperBPE 分词器³⁸，增加了子词级 LLM 每个 patch 的字节数。我们在 OLMo 分词器之上训练 SuperBPE 分词器，使词表大小达到 {200k, 400k}（k 表示千），使用与参考文献 ³⁸ 相同的 10-GB 文本样本进行分词器训练。我们使用 FOCUS³⁹ 初始化新 superword token 的嵌入。

我们的结果如图 2 所示。通过迁移到 SuperBPE，我们可以在很大程度上保持性能的同时加速子词级 LLM。然而，在某个词表大小阈值处，随着 softmax 的计算开销变得更大，子词级 LLM 会被帕累托支配。对于 OLMo 2 1B，这发生在词表大小介于 200k 和 400k token 之间时；尽管 softmax 分解^40,41 有可能避免这一点，但已观察到它们会导致收敛变慢和训练不稳定性增加²¹，并且需要新的嵌入初始化方法。字节级 LLM 不受 softmax 瓶颈的影响。这将使效率能够无限提升，而性能仅平滑下降。

有趣的是，BPE 合并优于熵合并和交叉熵合并，这与使用基于熵的块边界的早期工作^22,24相反。我们认为这可能是字节化设置所特有的模式，因为 BPE 合并策略为实现任意目标压缩率所做的不同合并次数最少（因此，在这个意义上，它最接近预训练模型）。

通过任务算术对字节化模型进行后训练

字节化为源 LLM 的生态系统增加了一个新组件（字节化模型）。一个自然的问题是：这个新组件如何与源 LLM 生态系统的其他组件交互？因此，我们研究了是否可以在不产生任何额外训练成本的情况下，将 Olmo 3 现有的后训练版本合并到 Bolmo 中以进行后训练。我们以直接通过强化学习在后训练中遵循指令的 Olmo 3 检查点（称为 RL-Zero²⁸）作为案例研究。我们发现，通过任务算术⁴²，将后训练检查点与基础 Olmo 3 的 transformer 层之间的权重差加到相应的 Bolmo 层上，就可以将指令遵循能力注入 Bolmo（图 3）。

尽管 Bolmo 基础模型在 IFEval 上的表现最初比 Olmo 3 差，但通过任务算术合并后，其性能提升到与原始后训练检查点相当的水平。我们得出结论：可以利用子词级 LLM 生态系统的组件来改进相应的字节化模型。这消除了子词级 LLM 后训练所极大受益的基础设施中对字节级 LLM 支持的前提条件^43,44，并大幅加快了迭代速度。

通过任务算术对字节化模型进行后训练时，一个微妙的要求是，后训练模型的嵌入值需要能够重置为基础模型嵌入的值，而不会造成大的性能损失，这通常是成立的，但并非总是如此（扩展数据图 2）。

两阶段训练的好处

两阶段训练增加了实现复杂度。我们进行了实验，直接从第 2 阶段开始，以测试这种额外的复杂度是否值得。将仅第 2 阶段训练与第 1 阶段 + 第 2 阶段训练进行公平比较是困难的：第 1 阶段训练所需的计算量更少（以浮点运算次数或 FLOPs 衡量），因为我们只通过全局模型的一小部分进行反向传播（方法中的“字节化流程”），并且内存效率更高，因为我们只需要存储一小部分优化器状态。我们通过大致匹配 FLOPs 并忽略内存不匹配来考虑这一差异。第 1 阶段大约需要 \(2\times {\mathrm{FLOPs}}_{{\mathcal{M}}}\)，而第 2 阶段大约需要 \(3\times {\mathrm{FLOPs}}_{{\mathcal{M}}}\)（通过全局模型的前向传播为 ×1，反向传播为 ×2）。因此，当省略第 1 阶段时，我们向第 2 阶段训练添加了 9.8B × 2/3 = 6.5B 个 token（将第 2 阶段的长度增加了 17%）。在实践中，我们认为 2/3 这个因子可能略微有利于仅第 2 阶段的运行，因为第 1 阶段的内存需求更低，并且可以使用针对推理的优化来加速第 1 阶段中使用的子词级 LLM 的前向传播。

图 4 比较了有第 1 阶段训练与无第 1 阶段训练的运行的训练轨迹。1B 模型从第 1 阶段训练中获益比 7B 更多，这表明更大的模型在直接开始第 2 阶段时可能对灾难性遗忘更具鲁棒性。每字节比特数差距在整个训练轨迹中缩小，但仍然有利于添加第 1 阶段。由于缺少第 1 阶段并未导致灾难性退化，我们认为一个合理的假设是，随着 token 预算的增加，第 1 阶段训练变得不那么重要；然而，这可能受到数据混合选择等因素的非平凡影响。第 1 阶段的一个关键好处是简化实验：快速获得一个应该接近子词级 LLM 性能的检查点，比反复运行完整训练实验创建了更短的反馈循环。

结论

我们已将 byteification 确立为在开发字节级 LLM 时从随机初始化开始训练的一个缺失方向。Byteification 让我们能够训练字节级 LLM，在 1B 和 7B 参数规模下性能接近最先进的子词级 LLM。这得益于专为 byteification 设计的架构和训练决策。Byteified 模型在推理速度上也接近子词级 LLM。我们还探索了字节级模型更高的灵活性，例如任意降低 token 粒度以实现更快的推理。Byteification 还让我们能够利用源子词模型周围生态系统的其他组件，一旦相应的基础模型完成 byteification，就可以以零样本模式对后训练模型进行 byteification。总体而言，byteification 可能使字节级 LLM 成为一种实用选择，其性能接近子词级 LLM 的水平，而资源投入仅为早期工作的一小部分。它们可能为 byteification 设置和字节级 LLM 从随机初始化开始训练开辟未来的研究方向（Supplementary Information 中的“Future directions”）。

方法

Byteified 模型架构细节

我们的架构遵循与早期 LTLM 相同的整体结构，包括 DTP、BLT 和 H-Net（Methods 中的“Related work”）。它由以下组件组成。

分词与嵌入

\({\mathcal{T}}\) 为 x 中的每个输入 UTF-8 字节（我们将 x 视为字节序列：x ∈ {0, …, 255}ⁿ）从包含每个字节条目的嵌入表中分配 \({{\mathbb{R}}}^{d}\) 中的对应嵌入。与子词嵌入表相比，字节嵌入表的大小可以忽略不计。然而，扩大嵌入表的规模和稀疏性已被证明可以提高性能，同时对推理速度没有负面影响⁴⁶。受 BLT^22,47 中哈希嵌入的启发，我们因此增大了嵌入表的大小。具体来说，我们将结束于当前字节位置的最长子词嵌入（来自子词级 LLM 中的原始嵌入表）残差地添加到每个字节嵌入中：

$${e}_{i}:= {{\mathcal{T}}}_{\mathrm{Byte}}({x}_{i})+{{\mathcal{T}}}_{\mathrm{SubwordSuffix}}({x}_{:i}),$$

其中 \({{\mathcal{T}}}_{\mathrm{SubwordSuffix}}\) 根据词汇表 \({{\mathcal{V}}}_{\mathrm{Subword}}\) 中与当前位置 i 之前的字节序列具有最长公共后缀的子词 token 的索引，为每个字节分配一个嵌入。例如，假设 {‘_flo’, ‘flo’, ‘lo’} 均不在原始子词词汇表中，则对应于 {‘_’, ‘_f’, ‘_fl’, ‘o’, ‘_flow’} 的子词嵌入可以被加到序列 {‘_’, ‘f’, ‘l’, ‘o’, ‘w’} 的字节表示上。保留子词嵌入并非严格必要，我们通常可以通过增大局部编码器的规模来达到相同的性能。然而，保留子词嵌入使我们能够通过增加廉价的稀疏激活参数数量，实现更好的性能–效率权衡；增大局部编码器规模和稀疏性的另一种替代方案是在前馈层中使用专家混合，不过我们在此不做研究。

局部编码器

局部编码器 \({\mathcal{E}}\) 通过一个 mLSTM 层³³ 对字节级嵌入进行上下文化，得到上下文化表示 \(\widehat{e}\)。我们发现，与其他线性循环神经网络（RNN）变体相比，mLSTM 提高了推理速度（扩展数据图 3），同时达到了有竞争力的性能。我们发现单个 mLSTM 层就足够了，因为保留的子词嵌入大幅增强了局部编码器的表达能力。

边界预测器

边界预测器 \({\mathcal{B}}\) 基于上下文化表示 \(\widehat{e}\) 为每个字节预测一个分数 p ∈ [0, 1]。如果 p 大于某个阈值，则在当前字节之后放置一个 patch 边界。与早期的 LTLM 不同，字节化模型的边界预测器是非因果的：它可以访问 1 个字节的未来上下文，并且仅用于预填充，此时可以利用未来信息，同时保留生成文本的能力。我们在“字节化语言模型架构”一节中详细描述了非因果边界预测，并在其中讨论了解码期间如何处理边界预测。

池化

We pool byte-level representations into patch representations by selecting the representation of the last byte in every patch as the patch-level representation h. This is equivalent to the pooling done by Hwang et al.² and does not introduce any extra parameters. Unlike Hwang et al.², the local models and the global model use the same representation dimensionality, obviating the need for an up-projection. We originally experimented with smaller local dimensions but found the up-projection mechanism to bottleneck performance by restricting the rank of the representations (Extended Data Fig. 1).

Global model

Most of the compute is spent in the deep global model \({\mathcal{M}}\) contextualizing the patch representations h into \(\widehat{h}\). We retain the global model of the original subword-level LLM.

Depooling

The global model is invoked at every patch boundary, providing contextualized representations for every patch. It remains to depool these representations back to representations of bytes. We do so by adding the latest available patch representation in \(\widehat{h}\) at any byte position to a linear projection of the byte representations \(\widehat{e}\), resulting in z. This is like the depooling by H-Net, again forgoing the projection due to equal global and local dimensionality.

Local decoder

The local decoder \({\mathcal{D}}\) contextualizes the depooled byte representations z into \(\widehat{z}\) through another stack of mLSTM layers. We used more mLSTM layers (in practice, four) to increase capacity, because unlike in the encoder, we found it infeasible to meaningfully reincorporate the output subword embedding matrix, which could have potentially allowed us to reduce the number of layers in the decoder in a similar way as for the encoder.

Language-modelling head

The language-modelling head converts the final byte representations \(\widehat{z}\) into scores interpretable as next-byte probabilities (and next-byte-followed-by-boundary probabilities if boundary predictions are fused; compare with ‘Byteified language model architecture’) through a projection to the vocabulary space and softmax. During decoding, the next atom (byte or boundary in the unfused case and byte or byte-and-boundary in the fused case) was generated by looping over the local encoder, local decoder and language-modelling head. If a boundary is predicted, the patch ends and is passed through the global model, after which the next loop over the local encoder, local decoder and language-modelling head starts. We did not find it necessary to employ any mechanism to ensure consistency between the patch representations and decoded bytes. As in standard subword-level language models, the extent of the possible mismatch between the hidden representations and the decoded atoms is dictated by the sampling procedure; any apparent mismatch is analogous to sampling a non-argmax token in a standard subword-level language model.

In practice, Bolmo 1B contains approximately 10M fewer parameters, where M is millions, than OLMo 2 1B (−0.7%), Bolmo 7B contains approximately 330M more than Olmo 3 7B (+4.5%), Blama 8B contains approximately 220M more than Llama 3 8B (+2.7%) and Bwen 8B contains approximately 120M more than Qwen3 8B (+1.5%).

Byteifying procedure

Subword-to-byte distillation

The first stage starts by initializing the parameters of the global model from the subword-level LLM checkpoint, whereas the parameters of the local models and the language-modelling head are initialized randomly. The aim of this stage is to quickly learn weights for the local encoder, local decoder, boundary predictor and language-modelling head that recover the behaviour of the subword model. Efficiency is crucial; the cost of this stage should be minimal to permit fast experimentation and allow increasing the investment into stage 2. To achieve these goals, we designed a stage 1 procedure that allows the model to learn the desired weights without fully backpropagating through the global model. This substantially reduces the time per training step (Extended Data Table 4). The stage 1 loss is minimal if and only if the byte-level model exactly mimics the source subword-level LLM. It is composed of three parts.

Quickly learning a boundary predictor \({\boldsymbol{\mathcal{B}}}_{{\bf{byteify}}}\)

We train the boundary predictor to emulate the boundaries placed by subword tokenization through a binary cross-entropy loss:

$${{\mathcal{L}}}_{{\mathcal{B}}}:= -\sum _{t}({{\mathcal{B}}}_{\mathrm{subword}}{(x)}_{t}\,\log \,{{\mathcal{B}}}_{\mathrm{byteify}}{(\hat{e})}_{t}+(1-{{\mathcal{B}}}_{\mathrm{subword}}{(x)}_{t})\,\log (1-{{\mathcal{B}}}_{\mathrm{byteify}}{(\hat{e})}_{t})),$$

where \({{\mathcal{B}}}_{\mathrm{subword}}(x)\) is 1 for every byte at the last position of a subword patch, otherwise 0. The boundary predictor \({{\mathcal{B}}}_{\mathrm{byteify}}\), which uses future context to tokenize the prefill text, quickly achieves over 99% accuracy.

Quickly learning a local encoder \(\boldsymbol{\mathcal{E}}\)

Assuming our boundary predictor perfectly emulates subword tokenization, our local encoder and pooling mechanism will be a perfect substitute for the subword embedding matrix if they yield the same input to the global model as the subword embedding matrix for every patch. This is the case if all pooled representations \(\mathrm{Pool}({\mathcal{E}}({\mathcal{T}}\,(x)),{{\mathcal{B}}}_{\mathrm{byteify}}(\hat{e}))\) are equal to the corresponding subword embeddings \({{\mathcal{T}}}_{\mathrm{subword}}(x)\). Hwang et al.² optimized towards this goal by directly minimizing the L² distance of every pooled representation to the corresponding subword embedding. We took an alternative approach inspired by research on model stitching, which showed that similar representations do not necessarily propagate through subsequent layers in a similar way⁴⁸. We propagate the pooled representations through n layers of the global model and minimize the L² distance to the subword representations that result from propagating the subword embeddings through the same n layers:

$$\begin{array}{l}{Y}_{{\mathcal{E}}}:= {{\mathcal{M}}}_{:n}({{\mathcal{T}}}_{\mathrm{subword}}(x)),\\ {\hat{Y}}_{{\mathcal{E}}}:= {{\mathcal{M}}}_{:n}(\mathrm{Pool}({\mathcal{E}}({\mathcal{T}}\,(x)),{{\mathcal{B}}}_{\mathrm{subword}}(x))),\\ {{\mathcal{L}}}_{{\mathcal{E}}}:= \parallel {Y}_{{\mathcal{E}}}-{\hat{Y}}_{{\mathcal{E}}}\parallel .\end{array}$$

where \({Y}_{{\mathcal{E}}}\) are the representations of the original model at layer n, \({\hat{Y}}_{{\mathcal{E}}}\) the retrofitted model representations at the same layer n of the global model, and \({{\mathcal{L}}}_{{\mathcal{E}}}\) the \({{\mathcal{L}}}^{2}\) distance between the two. Notably, we pool the local encoder representations using the true subword boundaries \({{\mathcal{B}}}_{\mathrm{subword}}\) instead of \({{\mathcal{B}}}_{\mathrm{byteify}}\); this is necessary to preserve the alignment of the pooled representations to the representations in \({{\mathcal{T}}}_{\mathrm{subword}}(x)\) along the sequence dimension. \({{\mathcal{M}}}_{:n}\) indicates the global model up to and including the nth layer. The weights of \({\mathcal{M}}\) are kept frozen. If n = 0, this reduces to the setting of ref. ². Although choosing n > 0 necessitates backpropagating through some parts of the global model, we can minimize the resulting cost by choosing a small n. We found that n = 4 strikes a good balance between performance and efficiency, as it substantially outperformed n = 0 while remaining cheap to compute.

Quickly learning a local decoder \(\boldsymbol{\mathcal{D}}\)

Our local decoder and language-modelling head are optimal if our byte-level LLM assigns the same likelihood as the subword model to every text x. Assuming equal patch boundaries, it is optimal if the likelihood of every patch is equal. As subword-level LLMs implicitly predict output patch boundaries, we cannot easily compute comparable patch likelihoods in byte-level models without output boundary prediction. In this case, we would have to resort to approximations, as in ref. ⁴⁹. However, as the byteified models do predict output patch boundaries, simply comparing the likelihoods of every patch results in an exact objective (proof in ‘Exactness of the stage 1 objective’ in Supplementary Information):

$$\begin{array}{l}{{\mathcal{L}}}_{{\mathcal{D}},\mathrm{Distill}}:= \sum _{i}f\left(\prod _{j\in T(x,i)}\,\mathrm{LMHead}\,({\hat{z}}_{\mathrm{subword}})[\,j,\mathrm{next}\_\mathrm{byte}(x,j)]\right.\\ \,\left.\parallel \,{\mathrm{LMHead}}_{\mathrm{subword}}({z}_{\mathrm{subword}})[i,\mathrm{next}\_\mathrm{tok}(x,i)]\right),\end{array}$$

where j ∈ T(x, i) indicates all byte indices j that are part of the ith subword patch; this includes the indices of the special <b> symbol if treated as separate or the indices of the 256 special symbols consisting of a byte plus <b> if fused. next_tok( ⋅ ) and next_byte( ⋅ ) map to the index in the vocabulary of the symbol occurring after the current symbol (token or byte), including special symbols. \({z}_{\mathrm{subword}}={\mathcal{M}}({{\mathcal{T}}}_{\mathrm{subword}}(x))\) are the representations of the subword model at the final layer, \({\hat{z}}_{\mathrm{subword}}={\mathcal{D}}(\mathrm{Depool}\,(\hat{e},{z}_{\mathrm{subword}},p))\) is the result of passing these representations through the depooling layer and the local decoder, and LMHead_subword is the language-modelling head of the source subword-level LLM. We chose for the comparison function f:

$$f(\hat{y}\,\parallel \,y):= -({y}^{1/\tau }\,\log \,{\hat{y}}^{1/\tau }+(1-{y}^{1/\tau })\,\log (1-{\hat{y}}^{1/\tau })),$$

where \(\widehat{y}\) are the predictions and y the targets. In principle, f could be any function that is minimal at \(\widehat{y}=y\); we chose the temperature-modulated binary cross-entropy with temperature τ = 5, as recommended by Minixhofer et al.⁴⁹. In practice, we conducted the operations involved in the loss computation in log-space to ensure stable numerics. We optionally combine the distillation loss \({{\mathcal{L}}}_{{\mathcal{D}},\mathrm{Distill}}\) with a cross-entropy loss to encourage the system to model the training data well and to start exploiting byte-level information, at the cost of giving up exactness if enabled:

$${{\mathcal{L}}}_{{\mathcal{D}},\mathrm{CE}}:= \sum _{j}-\log \,\mathrm{LMHead}\,({\hat{z}}_{\mathrm{subword}})[\,j,\mathrm{next}\_\mathrm{byte}(x,j)].$$

Putting it together

In principle, the boundary predictor and local encoder on the one hand and the local decoder and language-modelling head on the other could be trained separately (assuming we stop the gradient to the encoder through \({\widehat{z}}_{{\rm{subword}}}\)). Although there may be scenarios where this is beneficial, we chose to train them together for simplicity. The complete stage 1 loss is given by

$${{\mathcal{L}}}_{\mathrm{Stage1}}:= {\lambda }_{{\mathcal{B}}}{{\mathcal{L}}}_{{\mathcal{B}}}+{\lambda }_{{\mathcal{E}}}{{\mathcal{L}}}_{{\mathcal{E}}}+{\lambda }_{{\mathcal{D}},\mathrm{Distill}}{{\mathcal{L}}}_{{\mathcal{D}},\mathrm{Distill}}+{\lambda }_{{\mathcal{D}},\mathrm{CE}}{{\mathcal{L}}}_{{\mathcal{D}},\mathrm{CE}},$$

where \({\lambda }_{{\mathcal{B}}},{\lambda }_{{\mathcal{E}}},{\lambda }_{{\mathcal{D}},\mathrm{Distill}},{\lambda }_{{\mathcal{D}},\mathrm{CE}}\in {\rm{{\mathbb{R}}}}\) are the loss weights, which we set as follows: \({\lambda }_{{\mathcal{B}}}=4,{\lambda }_{{\mathcal{E}}}=1,{\lambda }_{{\mathcal{D}},\mathrm{Distill}}=1\,\mathrm{and}\,{\lambda }_{{\mathcal{D}},\mathrm{CE}}=1\). Stage 1 needs in total one forward pass through all layers and one backward pass through the first n layers of the global model plus forward and backward passes through local encoder, local decoder, boundary predictor and language-modelling head. This makes stage 1 substantially more efficient than training the entire model. It could also be further optimized by quantizing or applying inference-specific optimizations to the global model layers starting from the (n + 1)th layer (which we do not need to backpropagate through). We analysed the difference between inserting stage 1 and directly training the entire model end-to-end with randomly initialized parameters (besides the global model) in ‘Benefits of training in two stages’. Besides performance improvements, stage 1 provides a vehicle for rapid experimentation: we can conduct stage 1 training to rapidly check whether a particular architecture for the local encoder and decoder has sufficient capacity to emulate the input and output embedding matrices, respectively. We use this to guide the architecture search for our byteified models under the hypothesis that byte-level architectures that cannot emulate the subword model after stage 1 will remain inadequate with further stage 2 training.

End-to-end training

In the second stage, we train the entire model end-to-end, retaining only the boundary loss \({{\mathcal{L}}}_{{\mathcal{B}}}\) and the cross-entropy loss. For the cross-entropy loss (previously denoted \({{\mathcal{L}}}_{{\mathcal{D}},\mathrm{CE}}\)), we substitute the depooled representations \({\widehat{z}}_{{\rm{subword}}}\) computed from the subword model representations with the true depooled representations \(\widehat{z}\), and we refer to this new loss as \({{\mathcal{L}}}_{\mathrm{CE}}\) instead:

$${{\mathcal{L}}}_{\mathrm{Stage2}}:= {\lambda }_{{\mathcal{B}}}{{\mathcal{L}}}_{{\mathcal{B}}}+{\lambda }_{\mathrm{CE}}{{\mathcal{L}}}_{\mathrm{CE}}.$$

We now optimize all parameters, including those of the global model \({\mathcal{M}}\). This stage is intended for the model to adjust to the end-to-end setting, as in stage 1 we assumed a local encoder and boundary predictor perfectly emulating the subword model, which, although close, is not true in practice. The global model learns to exploit the new byte-level information in stage 2 and can optionally be trained with higher compression ratios of bytes per patch (section ‘Increasing model efficiency’).

Methods for increasing the compression factor

To increase the compression factor of a byteified model, we fix a compression ratio t for the target average bytes per patch. We then remove subword boundaries (merged subword tokens) of \({{\mathcal{B}}}_{\mathrm{subword}}\) until the desired compression ratio is achieved. We experimented with three merging strategies:

1. (1)

   BPE. We iteratively merged the most common pair of tokens as in byte pair encoding¹. In contrast to conventional BPE, we applied BPE per example instead of over the entire corpus. This was inspired by Feher et al.⁵⁰, who showed that it is possible to retrofit language models to operate over BPE merges of the tokens in their vocabulary.

2. (2)

   Entropy. We used an auxiliary 370M-parameter subword-level LLM, trained on 74.3B tokens following a downscaled version of the OLMo 2 training and architecture²⁹, to compute next-token entropies. We then iteratively merged the pair of patches that, when summing their individual entropies, resulted in the lowest entropy among all entropy sums of pairs of patches in the example.

3. (3)

   Cross-entropy. We used the same auxiliary LLM as for entropy-based merging, but instead of merging the pair of tokens with the lowest total entropy, we iteratively merged the pair of tokens with the lowest total cross-entropy with respect to the next token in the data.

For entropy- and cross-entropy-based merging, the auxiliary LLM was required during training time only to supervise the boundary predictor (as in DTP²⁴). Unlike BLT²², we did not need to retain the auxiliary LLM for inference.

Even though the loss is discontinuous with respect to the parameters of the boundary predictor and we did not employ any technique to backpropagate through the discrete boundary predictions, we observed stable training without loss spikes with all of the above merging methods. An important nuance is that the supervision target compression ratio t was not attained by the model. Despite the boundaries not being learned end to end, the model learns to trade off boundary prediction accuracy with the main next-byte prediction loss, like other multitask models that learn to balance performance on the constituent tasks (for example, ref. ⁵¹). An important hyperparameter is, thus, the factor \({\lambda }_{{\mathcal{B}}}\) controlling the importance of the boundary prediction task; we keep \({\lambda }_{{\mathcal{B}}}=4\) from stage 1 training.

Experimental set-up

Data

Our data mix consists of approximately 172B tokens, as tokenized by the Dolma2 Tokenizer (https://huggingface.co/allenai/dolma2-tokenizer) from the Dolma 3 pretraining data mix²⁸, augmented with 75M tokens of CUTE-style data¹¹, sampled so as not to overlap with the CUTE test set, to encourage character understanding (‘Character-level training data’ in Methods). Training ran for less than one epoch on this mix.

Model

We used Qwen3 8B Base³⁰, Llama 3 8B³¹ and the Olmo 3 7B checkpoint after mid-training and long-context extension²⁸ as the starting points for our byteified models. For the local models, we used stacks of alternating mLSTM³³ and feedforward layers of size 1 and 4 for the encoder and decoder, respectively (Extended Data Table 5).

Training

We trained stage 1 on 9.8B tokens (approximately 43B bytes). In this stage, we trained the local encoder, decoder, boundary predictor and language-modelling head, keeping the global model frozen. For stage 2, we trained the entire model on 39.3B tokens (approximately 173B bytes; Extended Data Table 4).

Evaluation

We created the 7B byteification evaluation suite based on the Olmo 3 OlmoBaseEval²⁸, skipping GSM Symbolic and BigCodeBench due to their size, and adding CUTE¹¹ and EXECUTE⁵² to measure character understanding in English and across other languages, respectively. We created the 1B byteification evaluation suite based on the Base Easy Suite in Olmo 3²⁸, again adding CUTE¹¹ to measure character understanding. For the 1B suite, we defined a set of core tasks consisting of ARC⁵³, MMLU⁵⁴, CSQA⁵⁵, HellaSwag⁵⁶, WinoGrande⁵⁷, SocialIQA⁵⁸, PiQA⁵⁹, the Basic Skills benchmark²⁸ and CUTE¹¹ for use in ablations and sweeps (Extended Data Tables 1 and 2). The 7B suite additionally comprises HumanEval⁶⁰, MBPP⁶¹, DS-1000 (ref. ⁶²), DeepSeek LeetCode⁶³, MultiPL-E⁶⁴, GSM8K⁶⁵, Minerva MATH⁶⁶, MedMCQA⁶⁷, MedQA⁶⁸, SciQ⁶⁹, DROP⁷⁰, Natural Questions⁷¹, SQuAD⁷², CoQA⁷³ and Lambada⁷⁴, in addition to the Jeopardy and Basic Skills tasks from OlmoBaseEval²⁸. We used PyTorch v.2.8 (ref. ⁷⁵) and vLLM v.0.11.0 (ref. ⁷⁶) to generate completions whenever available. Generation capabilities for the byteified models are implemented in our bolmo-core (https://github.com/allenai/bolmo-core) framework.

Character-level training data

To encourage models trained on our data mix to learn information about the characters within a word, we generated approximately 75M tokens (approximately 0.04% of the training data) for tasks requiring character-level understanding using the CUTE repository (https://github.com/Leukas/CUTE). Tasks included spelling out words, reversing words as well as swapping, deleting and substituting characters within a word given words in a source wordlist. We used a list of n = 150,000 words, ensuring zero overlap with the CUTE test words to avoid contamination. These data are purely in English. We did not use any multilingual character-understanding data, but we still observed large improvements on the multilingual EXECUTE benchmark, indicating that some texts requiring character-level understanding can help acquire generalizable knowledge about the characters within words. We observed that byte-level models otherwise do not acquire this knowledge through our short training schedule. However, training for longer, on more diverse data or with larger local models could act as alternative routes to acquiring character-level knowledge.

Statistical tests

To test whether model A statistically significantly outperforms model B, we started from the set of benchmark scores, D = {(a₁, b₁), (a₂, b₂), …, (a_n, b_n)}, where (a_i, b_i) are the scores of model A and model B on benchmark i. The set of benchmarks consists of all n tasks in the respective evaluation suite (the 7B byteification suite or 1B byteification suite with n = 40 and n = 22, respectively; Extended Data Tables 1 and 2). We then drew n pairs from D with replacement for every bootstrap iteration k ∈ {1, 2, …, N} with the number of bootstrap iterations N = 10,000:

$${D}_{k}^{\star }=\{({a}_{{j}_{1}},{b}_{{j}_{1}}),({a}_{{j}_{2}},{b}_{{j}_{2}}),\ldots ,({a}_{{j}_{n}},{b}_{{j}_{n}})\}.$$

We then computed the difference in performance for each bootstrap sample,

$${\delta }_{k}^{\star }=\frac{1}{n}\sum _{(a,b)\in {D}_{k}^{\star }}a-b,$$

resulting in an empirical distribution of performance differences \(\varDelta =\{{\delta }_{1}^{\star },{\delta }_{2}^{\star },\ldots ,{\delta }_{N}^{\star }\}\), which we used to compute an unadjusted P value:

$${p}^{{\rm{unadj}}}=\frac{1}{N}\mathop{\sum }\limits_{k=1}^{N}{\mathbb{1}}\,({\delta }_{k}^{\star }\le 0).$$

We then applied a Holm–Bonferroni correction for multiple comparisons. Given a total of K unadjusted P values {P^unadj(A, B), P^unadj(A, C), …, P^unadj(A, Z)}, we sorted the P values in ascending order and adjusted starting from i = 1 (the smallest P value), sequentially moving to larger P values:

$${p}_{(i)}^{{\rm{adj}}}=\min (\mathop{\max }\limits_{j\le i}((K-j+1){p}_{(j)}^{{\rm{unadj}}}),1).$$

Finally, we undid the sorting operation to arrive at an adjusted P value for every comparison.

Related work

Tokenization

LLMs process information represented as a discrete sequence of symbols called tokens or patches. The process of segmenting the input into this discrete sequence is called tokenization, with different ways to tokenize being used across modalities such as text¹⁰, audio⁷⁷ and images⁷⁸. The predominant approach for tokenizing text since the inception of LLMs has been subword tokenization^1,10: tokenizing text into a discrete sequence of units from a finite vocabulary of subword tokens (usually of size 30k–300k), typically represented as integer IDs. Subword tokenization causes several problems. (1) Information about the characters within each token is lost. Although LLMs have been shown to implicitly learn the constituent characters of their tokens^11,79 and although it is possible to explicitly re-introduce character information^12,80, they still fall short in tasks requiring character knowledge^11,13,81. (2) The implicit reliance of subword tokenization on the future contents of the text (called tokenization bias) causes unexpected behaviour at inference if the prompt ends in the middle of a word or with whitespace^18,19,20. (3) The need for a fixed, finite subword vocabulary causes restrictive rigidity: for example, although encoding English efficiently is crucial for pretraining, as the vast majority of current pretraining documents are in English, various downstream tasks have different efficiency requirements across different languages. (4) Tokenization in contemporary LLMs is tied to compute allocation: in a standard LLM, the same amount of compute is spent on processing every token in the prefill, every token contributes equally to the key–value cache (KV cache) size and a fixed amount of compute is spent on sequentially generating any new token. Although there are ways to mitigate this problem post hoc—such as KV cache sparsification⁸² and multi-token prediction⁸³—directly adapting the tokenization and, thus, the compute allocation based on the input instead might be more effective^22,24.

Byte-level LLMs

The shortcomings of subword tokenization have motivated extensive work on a wide range of alternatives, which even include tokenizing text by rendering it into pixels and segmenting these into patches^84,85,86. The most common alternative has been tokenizing into a smaller set of finer-grained atomic units, such as UTF-8 bytes. One strand of work directly replaces subword tokens with UTF-8 bytes, keeping other aspects of the architecture mostly the same^26,27,49,87. This can solve problems (1) and (2) of subword tokenization and potentially problem (3) with the right choice of fine-grained units^88,89. In any case, compute allocation remains a problem, exacerbated by having to process longer sequences. To mitigate this problem, some architectures pool a fixed number of tokens into a single representation with a lightweight local encoder (for example, another transformer network), pass the pooled representations through a deep global model operating over the shortened sequence and then depool the representations back to the original granularity through a local decoder. This approach has been pioneered for autoregressive models by the hourglass transformer⁹⁰ and later adopted more broadly^91,92,93. Recent subsequent work has shown that replacing static pooling with dynamic tokenization improves the performance–efficiency Pareto front^24,25. In this case, the token boundaries may be learned end to end, rely on entropy spikes or be externally supervised^2,24. We collectively refer to these architectures as LTLMs, as—although operating over bytes—they perform a tokenization step inside the model that aggregates the byte representations into representations over latent patches. Byte-level LTLMs can finally address issues (1) to (4) of subword tokenization. The most recent LTLMs have shown promise by performing on par with subword tokenization when spending the same overall number of FLOPs on training^2,22. Although we focused on LTLMs in this work, there are also other strands of promising research relevant to byte-level models, such as MrT5⁹⁴, which uses a soft gating mechanism to reduce sequence lengths at inference, and zip2zip⁹⁵, which adaptively merges tokens based on the past token context.

Tokenizer transfer and retrofitting

Techniques to alter the architecture of a model with extra training are typically referred to as retrofitting, which often relies on self-distillation^82,96. The principal difficulty when this involves a change of tokenizer is finding embeddings for the new tokens; this is usually done using heuristics^39,97,98 or training-based methods⁹⁹. Recently, effective tokenizer transfer methods based on cross-tokenizer distillation have been introduced^49,100,101. Here the original model is seen as the teacher, the tokenizer-transferred model is seen as the student and the objective is to match the behaviour of the student to the teacher. Byteification is a special case of tokenizer transfer. Byteification was first done by Pagnoni et al.²² by initializing the LTLM parameters from an existing subword model where possible and training as if from random initialization. Hwang et al.² later byteified by supervising the boundary prediction to match the subword boundaries and introducing an auxiliary embedding-matching loss. Our key contribution is creating an LTLM that is specifically suited to byteifying. We do so by introducing a new architecture (section ‘Byteified language model architecture’) and a dedicated two-stage procedure that efficiently byteifies by first learning to exactly recover the behaviour of the source subword model (section ‘Byteification training procedure’). Together, these innovations allow us to closely match the performance of state-of-the-art subword-level LLMs with a byteified model.

Data availability

All datasets used in this study are publicly available. Training data are available at https://huggingface.co/datasets/allenai/bolmo_mix. All evaluation data are available at GitHub (https://github.com/allenai/olmes), with the exception of the character-understanding datasets CUTE and EXECUTE, which are available at https://huggingface.co/datasets/leukas/cute and https://huggingface.co/datasets/benjamin/execute, respectively.

Code availability

The code needed to reproduce our results, including training code, is available at GitHub (https://github.com/allenai/bolmo-core). Our code is implemented in Python using PyTorch and based on the OLMo-core training framework, which is available at GitHub (https://github.com/allenai/olmo-core).

References

1. Sennrich, R., Haddow, B. & Birch, A. Neural machine translation of rare words with subword units. In Proc. 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Erk, K. & Smith, N. A.) 1715–1725 (Association for Computational Linguistics, 2016).

2. Hwang, S., Wang, B. & Gu, A. Dynamic chunking for end-to-end hierarchical sequence modeling. In Proc. Fourteenth International Conference on Learning Representations (eds Vondrick, C. et al.) 149273–149313 (International Conference on Learning Representations, 2026).

3. Brown, T. et al. Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems, Vol. 33 (eds Larochelle, H. et al.) 1877–1901 (Curran Associates, 2020).

4. Guo, D. et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 633–638 (2025).

   Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

5. Hofmann, V., Pierrehumbert, J. & Schütze, H. Superbizarre is not superb: derivational morphology improves BERT's interpretation of complex words. In Proc. 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (eds Zong, C. et al.) 3594–3608 (Association for Computational Linguistics, 2021).

6. Ahia, O. et al. Do all languages cost the same? Tokenization in the era of commercial language models. In Proc. 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 9904–9923 (Association for Computational Linguistics, 2023).

7. Land, S. & Bartolo, M. Fishing for magikarp: automatically detecting under-trained tokens in large language models. In Proc. 2024 Conference on Empirical Methods in Natural Language Processing (eds Al-Onaizan, Y. et al.) 11631–11646 (Association for Computational Linguistics, 2024).

8. Peng, Q., Chai, Y. & Søgaard, A. Understanding subword compositionality of large language models. In Proc. 2025 Conference on Empirical Methods in Natural Language Processing (eds Christodoulopoulos, C. et al.) 22524–22535 (Association for Computational Linguistics, 2025).

9. Zheng, B. S. et al. Broken tokens? Your language model can secretly handle non-canonical tokenizations. In Advances in Neural Information Processing Systems, Vol. 38 (eds Belgrave, D. et al.) 30322–30349 (Curran Associates, 2025).

10. Kudo, T. Subword regularization: improving neural network translation models with multiple subword candidates. In Proc. 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Gurevych, I. & Miyao, Y.) 66–75 (Association for Computational Linguistics, 2018).

11. Edman, L., Schmid, H. & Fraser, A. CUTE: measuring LLMs' understanding of their tokens. In Proc. 2024 Conference on Empirical Methods in Natural Language Processing (eds Al-Onaizan, Y. et al.) 3017–3026 (Association for Computational Linguistics, 2024).

12. Cosma, A., Ruseti, S., Radoi, E. & Dascalu, M. The strawberry problem: emergence of character-level understanding in tokenized language models. In Proc. 2025 Conference on Empirical Methods in Natural Language Processing (eds Christodoulopoulos, C. et al.) 28252–28263 (Association for Computational Linguistics, 2025).

13. Uzan, O. & Pinter, Y. CharBench: evaluating the role of tokenization in character-level tasks. In Proc. AAAI Conference on Artificial Intelligence, Vol. 40 (eds Koenig, S. et al.) 33296–33304 (AAAI Press, 2026).

14. Chirkova, N. & Troshin, S. CodeBPE: investigating subtokenization options for large language model pretraining on source code. In Proc. Eleventh International Conference on Learning Representations (eds Liu, Y. et al.) (International Conference on Learning Representations, 2023).

15. Zilio, L., Qian, S., Kanojia, D. & Orasan, C. Using character-level models for efficient abbreviation and long-form detection. In Proc. 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (eds Calzolari, N. et al.) 3028–3037 (ELRA and ICCL, 2024).

16. Dagan, G., Synnaeve, G. & Roziere, B. Getting the most out of your tokenizer for pre-training and domain adaptation. In Proc. 41st International Conference on Machine Learning, Vol. 235 (eds Salakhutdinov, R. et al.) 9784–9805 (PMLR, 2024).

17. Lindsey, L. M. et al. The impact of tokenizer selection in genomic language models. Bioinformatics 41, btaf456 (2025).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

18. Phan, B. et al. Exact byte-level probabilities from tokenized language models for fim-tasks and model ensembles. In Proc. Thirteenth International Conference on Learning Representations (eds Yue, Y. et al.) 38145–38166 (International Conference on Learning Representations, 2025).

19. Hayase, J., Liu, A., Smith, N. A. & Oh, S. Sampling from your language model one byte at a time. In Proc. Forty-third International Conference on Machine Learning (2026).

20. Vieira, T. et al. From language models over tokens to language models over characters. In Proc. Forty-second International Conference on Machine Learning, Vol. 267 (eds Singh, A. et al.) 61391–61412 (PMLR, 2025).

21. Liang, D. et al. XLM-V: overcoming the vocabulary bottleneck in multilingual masked language models. In Proc. 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 13142–13152 (Association for Computational Linguistics, 2023).

22. Pagnoni, A. et al. Byte latent transformer: patches scale better than tokens. In Proc. 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Che, W. et al.) 9238–9258 (Association for Computational Linguistics, 2025).

23. Yergeau, F. UTF-8, A Transformation Format of ISO 10646. Technical Report RFC 3629 (2003).

24. Nawrot, P., Chorowski, J., Lancucki, A. & Ponti, E. M. Efficient transformers with dynamic token pooling. In Proc. 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Rogers, A. et al.) 6403–6417 (Association for Computational Linguistics, 2023).

25. Slagle, K. SpaceByte: towards deleting tokenization from large language modeling. In Advances in Neural Information Processing Systems, Vol. 37 (eds Globerson, A. et al.) 124925–124950 (Curran Associates, 2024).

26. Wang, J., Gangavarapu, T., Yan, J. N. & Rush, A. M. MambaByte: token-free selective state space model. In Proc. First Conference on Language Modeling (2024).

27. Zheng, L. et al. EvaByte: Efficient Byte-level Language Models at Scale. HKU NLP Group https://hkunlp.github.io/blog/2025/evabyte (2025).

28. Olmo Team. Olmo 3. Preprint at https://doi.org/10.48550/arXiv.2512.13961 (2025).

29. Walsh, E. P. et al. 2 OLMo 2 furious (COLM's version). In Proc. Second Conference on Language Modeling (2025).

30. Yang, A. et al. Qwen3 technical report. Preprint at https://doi.org/10.48550/arXiv.2505.09388 (2025).

31. Grattafiori, A. et al. The Llama 3 herd of models. Preprint at https://doi.org/10.48550/arXiv.2407.21783 (2024).

32. Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30 (eds Guyon, I. et al.) (Curran Associates, 2017).

33. Beck, M. et al. xLSTM 7B: a recurrent LLM for fast and efficient inference. In Proc. Forty-second International Conference on Machine Learning, Vol. 267 (eds Singh, A. et al.) 3335–3357 (PMLR, 2025).

34. Minixhofer, B., Pfeiffer, J. & Vulić, I. CompoundPiece: evaluating and improving decompounding performance of language models. In Proc. 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 343–359 (Association for Computational Linguistics, 2023).

35. Fleshman, W. & Van Durme, B. Toucan: token-aware character level language modeling. Preprint at https://doi.org/10.48550/arXiv.2311.08620 (2023).

36. Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. The curious case of neural text degeneration. In Proc. International Conference on Learning Representations (eds Rush, A. et al.) (International Conference on Learning Representations, 2020).

37. Neitemeier, P., Deiseroth, B., Eichenberg, C. & Balles, L. Hierarchical autoregressive transformers: combining byte- and word-level processing for robust, adaptable language models. In Proc. Thirteenth International Conference on Learning Representations (eds Yue, Y. et al.) 51088–51111 (International Conference on Learning Representations, 2025).

38. Liu, A. et al. SuperBPE: space travel for language models. In Proc. Second Conference on Language Modeling (2025).

39. Dobler, K. & de Melo, G. FOCUS: effective embedding initialization for monolingual specialization of multilingual models. In Proc. 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 13440–13454 (Association for Computational Linguistics, 2023).

40. Morin, F. & Bengio, Y. Hierarchical probabilistic neural network language model. In Proc. Tenth International Workshop on Artificial Intelligence and Statistics, Vol. R5 (eds Cowell, R. G. & Ghahramani, Z.) 246–252 (PMLR, 2005).

41. Grave, É., Joulin, A., Cissé, M., Grangier, D. & Jégou, H. Efficient softmax approximation for GPUs. In Proc. 34th International Conference on Machine Learning, Vol. 70 (eds Precup, D. & Teh, Y. W.) 1302–1310 (PMLR, 2017).

42. Ilharco, G. et al. Editing models with task arithmetic. In Proc. Eleventh International Conference on Learning Representations (eds Liu, Y. et al.) (International Conference on Learning Representations, 2023).

43. Lambert, N. et al. Tulu 3: pushing frontiers in open language model post-training. In Proc. Second Conference on Language Modeling (2025).

44. Piché, A., Kamalloo, E., Pardinas, R., Chen, X. & Bahdanau, D. PipelineRL: faster on-policy reinforcement learning for long sequence generation. Trans. Mach. Learn. Res. https://openreview.net/forum?id=A35ak14Cyp (2026).

45. Zhou, J. et al. Instruction-following evaluation for large language models. Preprint at https://doi.org/10.48550/arXiv.2311.07911 (2023).

46. Huang, H. et al. Over-tokenized transformer: vocabulary is generally worth scaling. In Proc. 42nd International Conference on Machine Learning, Vol. 267 (eds Singh, A. et al.) 26261–26282 (PMLR, 2025).

47. Tito Svenstrup, D., Hansen, J. & Winther, O. Hash embeddings for efficient word representations. In Advances in Neural Information Processing Systems, Vol. 30 (eds Guyon, I. et al.) (Curran Associates, 2017).

48. Athanasiadis, I., Karmush, A. & Felsberg, M. Grounding functional similarity by invariance-aware model stitching. In Proc. Forty-third International Conference on Machine Learning (2026).

49. Minixhofer, B., Vulić, I. & Ponti, E. Universal cross-tokenizer distillation via approximate likelihood matching. In Advances in Neural Information Processing Systems, Vol. 38 (eds Belgrave, D. et al.) 79297–79326 (Curran Associates, 2025).

50. Feher, D., Vulić, I. & Minixhofer, B. Retrofitting large language models with dynamic tokenization. In Proc. 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Che, W. et al.) 29866–29883 (Association for Computational Linguistics, 2025).

51. Zhang, Y. & Yang, Q. A survey on multi-task learning. IEEE Trans. Knowl. Data Eng. 34, 5586–5609 (2022).

    Article  Google Scholar 

52. Edman, L., Schmid, H. & Fraser, A. EXECUTE: a multilingual benchmark for LLM token understanding. In Proc. Findings of the Association for Computational Linguistics: ACL 2025 (eds Che, W. et al.) 1878–1887 (Association for Computational Linguistics, 2025).

53. Clark, P. et al. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. Preprint at https://doi.org/10.48550/arXiv.1803.05457 (2018).

54. Hendrycks, D. et al. Measuring massive multitask language understanding. In Proc. International Conference on Learning Representations (ICLR) (eds Mohamed, S. et al.) (International Conference on Learning Representations, 2021).

55. Talmor, A., Herzig, J., Lourie, N. & Berant, J. CommonsenseQA: a question answering challenge targeting commonsense knowledge. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds Burstein, J. et al.) 4149–4158 (Association for Computational Linguistics, 2019).

56. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A. & Choi, Y. HellaSwag: Can a machine really finish your sentence? In Proc. 57th Annual Meeting of the Association for Computational Linguistics (eds Korhonen, A. et al.) 4791–4800 (Association for Computational Linguistics, 2019).

57. Sakaguchi, K., Le Bras, R., Bhagavatula, C. & Choi, Y. WinoGrande: an adversarial winograd schema challenge at scale. In Proc. AAAI Conference on Artificial Intelligence, Vol. 34 (eds Conitzer, V. & Sha, F.) 8732–8740 (AAAI Press, 2020).

58. Sap, M., Rashkin, H., Chen, D., Le Bras, R. & Choi, Y. Social IQa: commonsense reasoning about social interactions. In Proc. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (eds Inui, K. et al.) 4463–4473 (Association for Computational Linguistics, 2019).

59. Bisk, Y., Zellers, R., Le Bras, R., Gao, J. & Choi, Y. PIQA: reasoning about physical commonsense in natural language. In Proc. AAAI Conference on Artificial Intelligence, Vol. 34 (eds Conitzer, V. & Sha, F.) 7432–7439 (AAAI Press, 2020).

60. Chen, M. et al. Evaluating large language models trained on code. Preprint at https://doi.org/10.48550/arXiv.2107.03374 (2021).

61. Austin, J. et al. Program synthesis with large language models. Preprint at https://doi.org/10.48550/arXiv.2108.07732 (2021).

62. Lai, Y. et al. DS-1000: a natural and reliable benchmark for data science code generation. In Proc. Fortieth International Conference on Machine Learning, Vol. 202 (eds Krause, A. et al.) 18319–18345 (PMLR, 2023).

63. Guo, D. et al. DeepSeek-Coder: when the large language model meets programming—the rise of code intelligence. Preprint at https://doi.org/10.48550/arXiv.2401.14196 (2024).

64. Cassano, F. et al. MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation. IEEE Trans. Softw. Eng. 49, 3675–3691 (2023).

65. Cobbe, K. et al. Training verifiers to solve math word problems. Preprint at https://doi.org/10.48550/arXiv.2110.14168 (2021).

66. Lewkowycz, A. et al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, Vol. 35 (eds Koyejo, S. et al.) 3843–3857 (Curran Associates, 2022).

67. Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proc. Conference on Health, Inference, and Learning, Vol. 174 (eds Flores, G. et al.) 248–260 (PMLR, 2022).

68. Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11, 6421 (2021).

    Article  CAS  Google Scholar 

69. Welbl, J., Liu, N. F. & Gardner, M. Crowdsourcing multiple choice science questions. In Proc. 3rd Workshop on Noisy User-generated Text (eds Derczynski, L. et al.) 94–106 (Association for Computational Linguistics, 2017).

70. Dua, D. et al. DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds Burstein, J. et al.) 2368–2378 (Association for Computational Linguistics, 2019).

71. Kwiatkowski, T. et al. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguist. 7, 452–466 (2019).

    Google Scholar 

72. Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. 2016 Conference on Empirical Methods in Natural Language Processing, 2383–2392 (Association for Computational Linguistics, 2016).

73. Reddy, S., Chen, D. & Manning, C. D. CoQA: a conversational question answering challenge. Trans. Assoc. Comput. Linguist. 7, 249–266 (2019).

    Article  Google Scholar 

74. Paperno, D. et al. The LAMBADA dataset: word prediction requiring a broad discourse context. In Proc. 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Erk, K. & Smith, N. A.) 1525–1534 (Association for Computational Linguistics, 2016).

75. Ansel, J. et al. PyTorch 2: faster machine learning through dynamic Python bytecode transformation and graph compilation. In Proc. 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol. 2 (eds Abu-Ghazaleh, N. et al.) 929–947 (Association for Computing Machinery, 2024).

76. Kwon, W. et al. Efficient memory management for large language model serving with PagedAttention. In Proc. 29th Symposium on Operating Systems Principles (eds Flinn, J. et al.) 611–626 (Association for Computing Machinery, 2023).

77. Borsos, Z. et al. AudioLM: a language modeling approach to audio generation. IEEE/ACM Trans. Audio Speech Lang. Process. 31, 2523–2533 (2023).

    Article  Google Scholar 

78. Dosovitskiy, A. et al. An image is worth 16 × 16 words: transformers for image recognition at scale. In Proc. International Conference on Learning Representations (eds Mohamed, S. et al.) (International Conference on Learning Representations, 2021).

79. Kaushal, A. & Mahowald, K. What do tokens know about their characters and how do they know it? In Proc. 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Carpuat, M. et al.) 2487–2507 (Association for Computational Linguistics, 2022).

80. Xu, Z. et al. Enhancing character-level understanding in LLMs through token internal structure learning. In Proc. 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Che, W. et al.) 3839–3853 (Association for Computational Linguistics, 2025).

81. Hiraoka, T. & Inui, K. Spelling-out is not straightforward: LLMs' capability of tokenization from token to characters. In Proc. Findings of the Association for Computational Linguistics: EMNLP 2025 (eds Christodoulopoulos, C. et al.) 13340–13353 (Association for Computational Linguistics, 2025).

82. Łańcucki, A., Staniszewski, K., Nawrot, P. & Ponti, E. Inference-time hyper-scaling with KV cache compression. In Advances in Neural Information Processing Systems, Vol. 38 (eds Belgrave, D. et al.) 9365–9397 (Curran Associates, 2025).

83. Gloeckle, F., Youbi Idrissi, B., Roziere, B., Lopez-Paz, D. & Synnaeve, G. Better & faster large language models via multi-token prediction. In Proc. Forty-first International Conference on Machine Learning, Vol. 235 (eds Salakhutdinov, R. et al.) 15706–15734 (PMLR, 2024).

84. Lotz, J., Salesky, E., Rust, P. & Elliott, D. Text rendering strategies for pixel language models. In Proc. 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 10155–10172 (Association for Computational Linguistics, 2023).

85. Rust, P. et al. Language modelling with pixels. In Proc. Eleventh International Conference on Learning Representations (eds Liu, Y. et al.) (International Conference on Learning Representations, 2023).

86. Wei, H., Sun, Y. & Li, Y. DeepSeek-OCR: contexts optical compression. Preprint at https://doi.org/10.48550/arXiv.2510.18234 (2025).

87. Xue, L. et al. ByT5: towards a token-free future with pre-trained byte-to-byte models. Trans. Assoc. Comput. Linguist. 10, 291–306 (2022).

    Article  Google Scholar 

88. Limisiewicz, T., Blevins, T., Gonen, H., Ahia, O. & Zettlemoyer, L. MYTE: morphology-driven byte encoding for better and fairer multilingual language modeling. In Proc. 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Ku, L.-W. et al.) 15059–15076 (Association for Computational Linguistics, 2024).

89. Land, S. & Arnett, C. BPE stays on SCRIPT: structured encoding for robust multilingual pretokenization. Preprint at https://doi.org/10.48550/arXiv.2505.24689 (2025).

90. Nawrot, P. et al. Hierarchical transformers are more efficient language models. In Proc. Findings of the Association for Computational Linguistics: NAACL 2022 (eds Carpuat, M. et al.) 1559–1571 (Association for Computational Linguistics, 2022).

91. Yu, L. et al. MEGABYTE: predicting million-byte sequences with multiscale transformers. In Advances in Neural Information Processing Systems, Vol. 36 (eds Oh, A. et al.) 78808–78823 (Curran Associates, 2023).

92. Ho, N. et al. Block transformer: global-to-local language modeling for fast inference. In Advances in Neural Information Processing Systems, Vol. 37 (eds Globerson, A. et al.) 48740–48783 (Curran Associates, 2024).

93. Dolga, R., Maystre, L., Berariu, T. & Barber, D. From characters to tokens: dynamic grouping with hierarchical BPE. In Proc. Findings of the Association for Computational Linguistics: EMNLP 2025 (eds Christodoulopoulos, C. et al.) 11154–11162 (Association for Computational Linguistics, 2025).

94. Kallini, J., Murty, S., Manning, C. D., Potts, C. & Csordás, R. MrT5: dynamic token merging for efficient byte-level language models. In Proc. Thirteenth International Conference on Learning Representations (eds Yue, Y. et al.) 56646–56669 (International Conference on Learning Representations, 2025).

95. Geng, S. et al. zip2zip: inference-time adaptive tokenization via online compression. In Advances in Neural Information Processing Systems, Vol. 38 (eds Belgrave, D. et al.) 151818–151847 (Curran Associates, 2025).

96. Bick, A., Li, K. Y., Xing, E. P., Kolter, J. Z. & Gu, A. Transformers to SSMs: distilling quadratic knowledge to subquadratic models. In Advances in Neural Information Processing Systems, Vol. 37 (eds Globerson, A. et al.) 31788–31812 (Curran Associates, 2024).

97. Tran, K. From English to foreign languages: transferring pre-trained language models. Preprint at https://doi.org/10.48550/arXiv.2002.07306 (2020).

98. Minixhofer, B., Paischer, F. & Rekabsaz, N. WECHSEL: effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Proc. 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Carpuat, M. et al.) 3992–4006 (Association for Computational Linguistics, 2022).

99. Minixhofer, B., Ponti, E. M. & Vulić, I. Zero-shot tokenizer transfer. In Advances in Neural Information Processing Systems, Vol. 37 (eds Globerson, A. et al.) 46791–46818 (Curran Associates, 2024).

100. Dobler, K., Elliott, D. & de Melo, G. Token distillation: attention-aware input embeddings for new tokens. In Proc. Fourteenth International Conference on Learning Representations (eds Vondrick, C. et al.) 140590–140615 (International Conference on Learning Representations, 2026).

101. Haltiuk, M. & Smywiński-Pohl, A. Model-aware tokenizer transfer. Preprint at https://doi.org/10.48550/arXiv.2510.21954 (2025).

102. Shenfeld, I., Pari, J. & Agrawal, P. RL's razor: why online reinforcement learning forgets less. In Proc. Fourteenth International Conference on Learning Representations (eds Vondrick, C. et al.) 59839–59864 (International Conference on Learning Representations, 2026).

103. Gu, Y. et al. OLMES: a standard for language model evaluations. In Proc. Findings of the Association for Computational Linguistics: NAACL 2025 (eds Chiruzzo, L. et al.) 5020–5048 (Association for Computational Linguistics, 2025).

Download references

Acknowledgements

We thank the Beaker team at the Allen Institute for AI for providing and maintaining the training infrastructure, T. Romero for helpful discussions on inference efficiency, D. Heineman for help with the evaluation infrastructure, W. Merrill for useful discussions on linear recurrent neural networks, A. Liu for useful discussions on tokenization and D. Groeneveld for providing the checkpoint used for the entropy model. This work has been supported by the resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. We acknowledge the National Artificial Intelligence Research Resource pilot and Microsoft Azure for contributing to the results in this work. This research was supported with Cloud TPUs from Google’s TPU Research Cloud.

Funding

A.K discloses support for the research of this work from the UK EPSRC (EP/T02450X/1). N.A.S. discloses support for the research of this work from the National Science Foundation (Award No. 2413244). E.M.P. discloses support for the research of this work from the ERC (Starting Grant AToM-FM 101222956). The other authors received no specific funding for this work.

Author information

Authors and Affiliations

1. Allen Institute for AI, Seattle, WA, USA

   Benjamin Minixhofer, Tyler Murray, Noah A. Smith, Luca Soldaini & Valentin Hofmann

2. Language Technology Lab, University of Cambridge, Cambridge, UK

   Benjamin Minixhofer & Anna Korhonen

3. University of Washington, Seattle, WA, USA

   Tomasz Limisiewicz, Luke Zettlemoyer & Noah A. Smith

4. Imperial College London, London, UK

   Edoardo M. Ponti

5. LMU Munich, Munich, Germany

   Valentin Hofmann

6. Munich Center for Machine Learning, Munich, Germany

   Valentin Hofmann

Authors

1. Benjamin Minixhofer
2. Tyler Murray
3. Tomasz Limisiewicz
4. Anna Korhonen
5. Luke Zettlemoyer
6. Noah A. Smith
7. Edoardo M. Ponti
8. Luca Soldaini
9. Valentin Hofmann

Contributions

B.M., E.M.P., L.S. and V.H. conceived and designed the core experiments. B.M. performed the experiments. T.M. contributed to the experimental design and to the implementation and testing of the experimental set-up. T.L., A.K., L.Z. and N.A.S. provided guidance on the research direction, experiment design and paper writing. B.M., E.M.P., L.S. and V.H. wrote the original draft. All authors contributed to the review and editing of the Article.

Corresponding author

Correspondence to Benjamin Minixhofer.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature thanks Xinyi Wang, Yingfei Xiong, Zhao Zhang and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Explained variance ratio of the singular values of the input and output embedding matrices.

a-d, Input embedding explained variance ratios acros model families. e-h, Output embedding explained variance ratios across model families. Variance ratios are normalized by the number of dimensions; they decay smoothly with the number of components, up until a steep dropoff toward the highest-rank components. This indicates that it is difficult to approximate the embedding matrices using lower-rank structure. A notable exception is Qwen3-4B-Base, which may be more amenable to a lower-dimensional local encoder.

Extended Data Fig. 2 Embedding resettability across models.

Since we only have a one-to-one correspondence between the parameters of the source subword-level LLM and the parameters of the global model \({\mathcal{M}}\), we can only easily adapt \({\mathcal{M}}\) via task arithmetic. The local encoder \({\mathcal{E}}\) and decoder \({\mathcal{D}}\) remain in the base model space. Whether the post-training transfer is successful thus depends on whether the base input embedding space and the base output embedding space remain compatible with post-trained inner Transformer layers; that is, on whether resetting the embeddings of the post-trained model to the base model embeddings preserves performance. a, Cross-entropy loss for post-trained models, and the same post-trained models with the embeddings reset to the corresponding base model embeddings; loss is computed on examples from the Tulu 3 dataset⁴³. b, Number of model parameters vs. the loss ratio of the model with reset embeddings to the original post-trained model. The number of parameters explains some variance (with larger models being more amenable to reset embeddings), and models post-trained via reinforcement learning (the Olmo 3 RL-Zero family²⁸) are more amenable to embedding resetting; this is in line with the findings of Shenfeld et al.¹⁰².

Extended Data Fig. 3 Inference efficiency measurements.

a, Greedy decoding throughput (bytes/s) across compression factors and prefill lengths. b, Prefilling latency (time to first byte) across compression factors and prefill lengths; Bolmo 7B overtakes Olmo 3 7B at a compression of ~6.6 bytes per patch. c, Decoding throughput for 18.0K prefill bytes across candidate local model architectures and of the final chosen architecture. d, Prefilling latency for the same candidate models and the final chosen architecture. All numbers recorded at batchsize  =  1 on H100 GPUs. Non-greedy sampling affects throughputs by <  5% in our experiments. Shaded areas indicate one standard deviation.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Cite this article

Minixhofer, B., Murray, T., Limisiewicz, T. et al. Retrofitting language models to operate over bytes. Nature (2026). https://doi.org/10.1038/s41586-026-11111-4

Download citation

• Received: 13 February 2026

• Accepted: 01 September 2026

• Published: 07 October 2026

• Version of record: 07 October 2026

• DOI: https://doi.org/10.1038/s41586-026-11111-4