arXiv cs.AI· Ruize Xu, Chenxiao Yang, Yanhong Li, David McAllester·· 14 小时前AI 评分39
训练 Transformer 成为通用计算机：小模型学会执行 MicroPy 程序
Towards Training Transformers as a Universal Computer
AI 导读
一项 EMNLP 2026 研究发现，小型 Transformer 经随机生成的 MicroPy 程序训练后，能借助 PENCIL 脚手架在有限上下文窗口内预测小步执行过程，并泛化到比特复制与翻转、二进制加减乘、SAT 验证与求解等未见人类手写程序。由于 MicroPy 具备计算通用性，该结果为标准 Transformer 可被训练为通用计算机提供了实证。
正文 · 原文
Abstract:We demonstrate that a small transformer can learn to execute programs in MicroPy, a simplified yet computationally universal programming language. Given procedure definitions together with an expression to evaluate, the transformer predicts small-step execution using PENCIL scaffolding for space-efficient execution within a bounded context window. After training on randomly generated, semantically arbitrary MicroPy programs, the learned transformer generalizes to held-out human-written programs including bit copying and flipping, binary addition and multiplication, and SAT verification and solving. The trained model exhibits out-of-distribution generalization: it correctly evaluates programs drawn from a substantially different distribution than the one used to generate its training data. Since MicroPy can express any computation, our results provide empirical evidence that a standard transformer can be trained to act as a universal computer when paired with PENCIL scaffolding. Project page: this https URL
| Comments: | Findings of EMNLP 2026. 22 pages, 5 figures, 3 tables. Project page: this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.25166 [cs.AI] |
| (or arXiv:2604.25166v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.25166 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: EMNLP 2026, pages 48933-48954 |
Submission history
From: Ruize Xu [view email]
[v1]
Tue, 28 Apr 2026 03:15:44 UTC (626 KB)
[v2]
Thu, 29 Oct 2026 10:55:24 UTC (330 KB)
来源：arXiv cs.AI · arxiv.org