Hacker News· sinnsro·· 6 天前AI 评分32
Byte Language Models：扩展性、涌现抽象与信息分配
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
AI 导读
研究显示，无分词器的字节级 Transformer 在模型规模扩大时持续优于子词 Transformer，并能自发学到类似分词的局部文本抽象结构。将最多 25% 的中间层限制在这些局部表示上仍可保持下游性能，且生成不确定性集中在局部结构边界，用于投机解码时被接受的 token 数比子词 Transformer 多 3.4 倍。
正文 · 原文
Abstract:Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to $25\%$ of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields $3.4\times$ more accepted tokens than in subword Transformers.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.05978 [cs.CL] |
| (or arXiv:2610.05978v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.05978 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jie Wang [view email]
[v1]
Mon, 5 Oct 2026 08:29:08 UTC (479 KB)
来源：Hacker News · arxiv.org