arXiv cs.CL· Giovanni Monea, Nathan Godey, Kiant\'e Brantley, Yoav Artzi·· 12 小时前AI 评分38
状态-预测分离假设：Transformer 用独立计算流分离状态存储与下一 token 预测
The State-Prediction Separation Hypothesis
AI 导读
研究者提出“状态-预测分离假设”，认为将 Transformer 中“预测下一 token”与“为未来预测存储状态”两种角色解耦可提升语言建模性能。他们设计了一种使用独立计算流的 Transformer 变体，并在多种规模下开展预训练实验，结果显示该设计在数据与计算效率上持续更优，下游任务平均比标准 Transformer 高 2-3 个百分点。
正文 · 原文
Abstract:Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the state-prediction separation hypothesis: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses separate computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data- and compute-efficiency, improving validation loss and outperforming standard Transformers by 2-3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails.
| Comments: | Published at NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.01218 [cs.CL] |
| (or arXiv:2607.01218v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01218 arXiv-issued DOI via DataCite |
Submission history
From: Giovanni Monea [view email]
[v1]
Wed, 1 Jul 2026 17:55:09 UTC (1,510 KB)
[v2]
Thu, 5 Nov 2026 16:43:45 UTC (2,416 KB)
来源：arXiv cs.CL · arxiv.org