arXiv cs.AI· Hyunjong Ok, Jaeho Lee·· 4 小时前AI 评分34
无需 Transformer:LLM 直接编写权重时更倾向稀疏表结构
Weight-Space Programming Doesn't Need Transformers
AI 导读
研究让三个前沿 LLM 在无梯度训练条件下直接生成权重或自行设计架构,结果三者均将纯文本事实编译为稀疏表并用固定流程查询,测试准确率最高达 54.4%。LLM 自行设计的模型准确率超过同一 LLM 手写权重的最准 Transformer,参数量少约 1,800 倍。作者据此提出基于联想记忆的模型,在 GPU 上比同尺寸 Transformer 快最多 17 倍。
正文 · 原文
Abstract:Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves. When free to design, all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An LLM-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800x fewer parameters. Without gradients, they choose a table, in which the LLM can state what each weight means and can find and rewrite a wrong one. Each wrong answer points to one row, and removing a row changes about one other answer. From this, we propose an associative-memory-based model up to 17x faster on GPUs than a same-sized Transformer. We also test pretrained weights, other benchmarks, and tasks where models generate answers. Pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights.
| Comments: | Preprint (Work in progress) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.22055 [cs.LG] |
| (or arXiv:2611.22055v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22055 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hyunjong Ok [view email]
[v1]
Tue, 17 Nov 2026 01:33:13 UTC (253 KB)
来源:arXiv cs.AI · arxiv.org