arXiv cs.CL· Hao Wu·· 13 小时前AI 评分36
Delta N-gram：向语言模型原生 N-gram 记忆的前向写入方法
Delta N-gram: Forward-Only Continual Writing into a Language Model's Native N-gram Memory
AI 导读
Delta N-gram 提出一种仅需前向传播的写入路径，预训练哈希 N-gram 表 T^0 保持不变，证据写入稀疏补丁 Δ，由 4.1M 参数写入器为每个（位置、哈希头）事件生成目标与门控，清除 Δ 即可恢复原模型。
正文 · 原文
Abstract:Our previous analysis found that a model's native hashed N-gram table is where continual updates disturb the base least, but its writes needed backpropagation and changed the pretrained table. We present Delta N-gram, a forward-only write path into that table. The pretrained table T^0 is never modified. Evidence is written into a sparse patch \Delta on the rows it addresses: a 4.1M-parameter writer emits a target and a gate per (position, hash head) event, and a block-averaged delta rule keeps each row a convex combination of its old value and the gated targets. The writer is trained offline on raw text with the pretraining objective; at use time writing takes forward passes only, and clearing \Delta restores the original model. A 0.6B prototype trained to answer questions keeps 16/16 facts after a second write; online gradient writes keep 2--4/16. We study two writers: a plain writer that encodes evidence with the T^0-only model, and a delta writer that reads its current memory while encoding. On Qwen3.8-Flash-Next, the plain writer's memory is document-specific: on 64 held-out news events it lowers continuation NLL by 0.068 nats/token against empty memory and by 0.024 against another document's memory, and 0.063 and 0.013 after 16 intervening articles. The delta writer's memory is more document-specific and keeps more of it under distraction (70% vs 54%). Over 35,000 writes on our previous seven-task chain both writers stay stable and the delta writer drifts less (layer-2 math drift 176% vs 188%); later accuracy changes by -0.26/-0.52pp (-7.29pp for gradient updates) and small-k_i drift is 17.9/17.5% (39.2%). Acquisition is negative, and general-input drift reaches 32.8% (20.75%): format rows read by every example grow to 4--12 times their pretrained norm. Zeroing eight such rows removes over 98% of layer-2 drift on math inputs, together with their NLL gain.
| Comments: | 33 pages, 3 figures, 9 tables, 2 algorithms. Follow-up to arXiv:2610.14921. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.31125 [cs.CL] |
| (or arXiv:2610.31125v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31125 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hao Wu [view email]
[v1]
Thu, 22 Oct 2026 11:42:49 UTC (457 KB)
来源：arXiv cs.CL · arxiv.org