arXiv cs.CL· Nicol\'as Vera Z\'u\~niga·· 2 天前AI 评分31
字节级模型该在哪里分配算力：BLT 的 novelty patching 压缩视角
Patch where the text is new: a compression view of where byte-level models should spend compute
AI 导读
针对 Byte Latent Transformer（BLT）按下一字节熵值划分 patch 会跳过算式结果的问题，研究者提出基于 LZ77 压缩思想的 novelty patching：在字节不太可能重复近期上下文处开启 patch，该规则仅需一张基于前两字节的查找表，数秒内即可在无标签原始文本上拟合完成。
正文 · 原文
Abstract:Byte-level language models such as the Byte Latent Transformer (BLT) run their large global model once per patch and start a patch where a small model's next-byte entropy is high. This rule rarely starts a patch at in-line computed results, such as the number after "=" in a worked solution, and accuracy on them collapses under tight patch budgets. We propose novelty patching, motivated by LZ77-style compression: start a patch where the upcoming bytes are unlikely to repeat the recent context. The rule is a lookup table over the previous two bytes, fitted in seconds on raw text with no labels, reference model or losses. In BLT-style models trained from scratch on a 1.9 GB math corpus at a matched ~11% patch rate (3 seeds each), it has no clear effect at 1M parameters, raises final-answer accuracy from 47% to 72% at 12M, and at 50M raises computed-result accuracy from 21% to 57% and final-answer accuracy from 60% to 90%; a table fitted on the raw pretraining corpus keeps the gain. Outside math the advantage shrinks: novelty lowers bits per byte on prose and ties entropy on source code and on synthetic execution traces. An analysis of where boundaries fall explains the pattern. Computed results have ordinary surprisal for the patcher's model, so surprisal-seeking rules pass over them: entropy concentrates patch starts on high-surprisal bytes most strongly of all rules (boundary enrichment 1.61) and is the worst on computed results, while the best rules sit near chance enrichment. Novelty works where new content begins at segment starts close to the computation, as in written math; a learned chunker (H-Net) finds such structure on its own, in math and in code. We also report what did not work: typed uncertainty as a training rule, switching rules after training, and a context-aware novelty predictor.
| Comments: | 10 pages, 2 figures, 5 tables. Code, logs and results: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.22100 [cs.CL] |
| (or arXiv:2610.22100v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22100 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nicolás Vera [view email]
[v1]
Thu, 15 Oct 2026 15:47:08 UTC (285 KB)
来源：arXiv cs.CL · arxiv.org