arXiv cs.CL· Tianyi Zhang, Feiying Chen·· 9 小时前AI 评分31
逐 token 训练下的潜在事实组合:Baton 系列循环 Transformer 研究
Latent Fact Composition under Next-Token Training
AI 导读
研究构建合成语料,考察逐 token 训练的模型能否回答中间实体未出现在任何两跳训练样本中的两跳问题。标准 Transformer、循环 Transformer 和 DiscoLoop 几乎全部答错,新提出的 Baton 系列循环 Transformer 将中间实体以 token 嵌入传入第二次循环,仅预测答案时三个变体中两个近乎全对。
正文 · 原文
Abstract:Controlled studies of latent two-hop reasoning train transformers to predict only the answer, whereas language models are trained to predict every token. We ask whether a model trained on every token learns to answer two-hop questions whose intermediate entity appears in no two-hop training example. We build a synthetic corpus with such questions. On this corpus every term of the loss can be computed exactly. A looped transformer, a standard transformer and DiscoLoop answer almost none of these questions. We therefore introduce the Baton family of looped transformers. They pass the intermediate entity to the second loop as a token embedding. When trained to predict only the answer, two of its three variants answer these questions almost perfectly. We then relate the success and failure of each model to three design conditions. Experiments that change one condition at a time agree with this hypothesis. When Baton is trained on every token, however, its accuracy on these questions rises and then declines. We further split the gradient of the next-token loss into three terms. Removing the two terms that depend on the particular training set raises the peak accuracy but does not remove the decline. The decline arises in the second hop while the model still recalls both facts.
| Comments: | 21 pages, 5 figures, 23 tables |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| ACM classes: | I.2.6; I.2.7 |
| Cite as: | arXiv:2611.21921 [cs.CL] |
| (or arXiv:2611.21921v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21921 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianyi Zhang [view email]
[v1]
Mon, 16 Nov 2026 21:33:27 UTC (449 KB)
来源:arXiv cs.CL · arxiv.org