Hacker News· E-Reverance·· 8 天前AI 评分27
Residual Matrix Transformers：扩大残差流规模的 RMT 架构
Residual Matrix Transformers: Scaling the Size of the Residual Stream
AI 导读
研究者提出 Residual Matrix Transformer（RMT），用外积记忆矩阵替换 Transformer 的残差流，使残差流规模可与计算量和模型规模独立扩展。RMT 以少 58% FLOPS、25% 参数和 41% 训练 token 达到与 Transformer 相同的损失，并在下游评测中表现更优。该工作已被 ICML 2025 接收，代码已开源。
正文 · 原文
Abstract:The residual stream acts as a memory bus where transformer layers both store and access features (Elhage et al., 2021). We consider changing the mechanism for retrieving and storing information in the residual stream, and replace the residual stream of the transformer with an outer product memory matrix (Kohonen, 1972, Anderson, 1972). We call this model the Residual Matrix Transformer (RMT). We find that the RMT enjoys a number of attractive properties: 1) the size of the residual stream can be scaled independently of compute and model size, improving performance, 2) the RMT can achieve the same loss as the transformer with 58% fewer FLOPS, 25% fewer parameters, and 41% fewer training tokens tokens, and 3) the RMT outperforms the transformer on downstream evaluations. We theoretically analyze the transformer and the RMT, and show that the RMT allows for more efficient scaling of the residual stream, as well as improved variance propagation properties. Code for this project can be found at this https URL.
| Comments: | Accepted to ICML 2025 |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2506.22696 [cs.LG] |
| (or arXiv:2506.22696v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.22696 arXiv-issued DOI via DataCite |
Submission history
From: Brian Mak [view email]
[v1]
Sat, 28 Jun 2025 00:29:42 UTC (1,898 KB)
来源：Hacker News · arxiv.org