Scratchpad Patching：在字节级语言模型中解耦计算量与 patch 大小

View PDF HTML (experimental)

Abstract:Tokenizer-free language models eliminate the tokenizer step of the language modeling pipeline by operating directly on bytes; patch-based variants further aggregate contiguous byte spans into patches for efficiency. However, the average patch size chosen at the model design stage governs a tight trade-off: larger patches reduce compute and KV-cache footprint, but degrade modeling quality. We trace this trade-off to patch lag: until a patch is fully observed, byte predictions within it must rely on a stale representation from the previous patch to preserve causality; this lag widens as patches grow larger. We introduce Scratchpad Patching (SP), which inserts transient scratchpads inside each patch to aggregate the bytes seen so far and refresh patch-level context for subsequent predictions. SP triggers scratchpads using next-byte prediction entropy, selectively allocating compute to information-dense regions and enabling post-hoc adjustment of inference-time compute. Across experiments on natural language and code, SP improves model quality at the same patch size; for example, even at $16$ bytes per patch, SP-augmented models match or closely approach the byte-level baseline on downstream evaluations while using a $16\times$ smaller KV-cache over patches and requiring roughly $3$-$4\times$ less inference compute than the byte-level baseline.

Comments:	31 pages, 15 figures
Subjects:	Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:	arXiv:2605.09630 [cs.CL]
	(or arXiv:2605.09630v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2605.09630 arXiv-issued DOI via DataCite

Submission history

From: Lin Zheng [view email]
[v1] Sun, 10 May 2026 16:18:22 UTC (195 KB)
[v2] Fri, 6 Nov 2026 17:01:55 UTC (252 KB)