arXiv cs.CL· Nicol\'as Vera Z\'u\~niga·· 9 天前AI 评分22
BLT 的熵触发分块存在盲区：边界依赖信号能找回被跳过的计算结果
Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching misses
AI 导读
字节级模型 BLT 按下一字节熵值决定 patch 起点，会系统性跳过"类型可预测但数值需计算"的位置，如数学解题中"="后的结果。在 Meta BLT-1B 上，10% 字节设 patch 起点时熵规则仅答对 19.0% 的计算结果，每个"="后设边界可达 51.8%，熵结合无标签的边界依赖信号达 67.1%。该差距在低秩微调后仍存在（32.9% vs 72.7%），并随模型规模扩大而增大。
正文 · 原文
Abstract:Byte-level language models such as the Byte Latent Transformer (BLT) group bytes into patches and run their large global model once per patch. BLT starts a patch where a small model's next-byte entropy is high, so global compute goes where the next byte is hard to predict. We show that this rule has a systematic blind spot: positions whose type is predictable but whose value must be computed, such as the number after "=" in a worked math solution. Under tight patch budgets, entropy-triggered layouts skip these positions, and accuracy on them collapses. In Meta's BLT-1B with patch starts on 10% of bytes, the entropy rule puts a patch start at 16% of the computed results in GSM8K solutions and gets 19.0% of them exactly right; a boundary after each "=" at the same patch count gets 51.8%, and entropy combined with a label-free boundary-dependence signal gets 67.1% (default layout at 26% of bytes: 76.8%). The gap survives adapting BLT-1B to the budget with low-rank fine-tuning (32.9% vs 72.7%, three runs per rule, paired p < 1e-200) and grows with model size in byte models trained from scratch at a 10% budget: at 1M, 12M and 50M parameters, boundary dependence beats entropy on final answers by -1.6, +10.1 and +19.8 points, and at 50M it gets 35.9% of computed results against 13.9% (3 seeds each). BLT's entropy-jump rule helps neither target at 50M. The entropy trigger of Scratchpad Patching is likewise indistinguishable from random scratchpads on final answers (5.6% vs 5.9%, 5 seeds), while answer-start scratchpads give 38.1%. The effect is specific to computed values: copies and lookups gain little, and values the model cannot compute gain nothing. Boundary dependence, the rise in the model's own loss when a patch start is removed, measured per two-byte context, finds these positions without labels: combined with entropy it beats the hand-written rule on computed results.
| Comments: | 13 pages, 4 figures, 6 tables. Code, logs and results: this https URL (archived: doi:https://doi.org/10.5281/zenodo.23238215) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.11790 [cs.CL] |
| (or arXiv:2610.11790v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11790 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nicolás Vera [view email]
[v1]
Thu, 8 Oct 2026 12:03:45 UTC (66 KB)
来源：arXiv cs.CL · arxiv.org