arXiv cs.CL· Yuhe Hu·· 12 小时前AI 评分31
训练后 TTT 层中的误差修正机制分析
Error Correction in Trained TTT Layers
AI 导读
研究分析了已发布 TTT 实现中预测驱动更新的结构与作用，发现移除归一化预测贡献后，TTT-Linear-760M 和 1.3B 在《战争与和平》100,303 个 token 上困惑度仅上升 2.8-2.9%，而冻结全部 fast-weight 更新则使困惑度上升 4.2-4.5 倍。
正文 · 原文
Abstract:Test-time-training (TTT) layers update fast-weight memory while processing an input sequence. A gradient-based update, however, does not by itself establish how the memory revises an existing association. We study the structure and empirical role of prediction-derived updates in released TTT implementations. For dot-product objectives, we characterize the absence of a prediction-error residual and establish conditions under which inner-loop ascent and descent are equivalent after reparameterization. For the LayerNorm-based squared-error objective used by TTT-Linear and TTT-MLP, we derive the inner gradient and identify a prediction-derived component that cancels at the released initialization in the zero-epsilon limit. We then measure the full normalized prediction's contribution on three trained checkpoints. Removing this contribution while retaining the LayerNorm Jacobian gives a mean gradient norm ratio of 0.25 to 0.32 across the models. On 100,303 tokens from War and Peace, the same intervention with local per-token norm matching raises perplexity by 2.8-2.9% on TTT-Linear-760M and 1.3B, whereas freezing all fast-weight updates raises it by factors of 4.2-4.5. Separate synthetic tests illustrate the dependence of plain delta-rule overwriting on step size. The results distinguish a prediction-derived update from its behavioral value and show why implementation analysis and live interventions are both needed to assess test-time memory.
| Comments: | 13 pages, 3 figures. Code and saved results included as ancillary material and at this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.16268 [cs.LG] |
| (or arXiv:2610.16268v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16268 arXiv-issued DOI via DataCite |
Submission history
From: Yuhe Hu [view email]
[v1]
Mon, 12 Oct 2026 06:48:31 UTC (297 KB)
[v2]
Fri, 16 Oct 2026 04:46:07 UTC (298 KB)
来源：arXiv cs.CL · arxiv.org