arXiv cs.AI· Davide Venuto, Leonardo Venuto·· 14 小时前AI 评分29
GPT-2 类模型的隐式推理：通过训练分布设计弥合组合性差距
Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models
AI 导读
一项受控实验用从零训练的 GPT-2 类 Transformer 研究两跳推理中的组合性差距，发现该差距并非架构固有限制。研究者在不改架构、不加 OOD 组合监督的前提下，将 ID 与 OOD 原子事实按与组合电路第二跳检索位置对齐的格式呈现，模型成功把已学的 ID 组合行为迁移到留出的 OOD 原子事实上，弥合了差距。
正文 · 原文
Abstract:Compositional generalization remains a persistent limitation of transformer language models: even when a model learns to combine known facts correctly in-distribution (ID), the same compositional mechanism may fail when applied to out-of-distribution (OOD) knowledge. We investigate this compositionality gap in a controlled two-hop reasoning setting using a GPT-2-like transformer model trained from scratch. Building on prior work identifying a circuit associated with compositional generalization, we test whether the failure of OOD transfer is related to how atomic knowledge is structurally exposed during training. We modify the training distribution by presenting both ID and OOD atomic facts in a format positionally aligned with the second-hop retrieval step of the learned compositional circuit, without modifying the model architecture or adding OOD compositional supervision. Under this intervention, the model successfully transfers the learned ID compositional behaviour to held-out OOD atomic facts, closing the observed compositionality gap in the controlled setting. Our results provide evidence that the observed compositionality gap is not a fundamental architecture limitation and that it can be overcome through mechanistically informed training distribution design.
| Comments: | Code available at: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39016 [cs.CL] |
| (or arXiv:2610.39016v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39016 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Davide Venuto [view email]
[v1]
Wed, 28 Oct 2026 22:44:54 UTC (414 KB)
来源：arXiv cs.AI · arxiv.org