arXiv cs.MA· Sergey Salishev, Zeev Volkovich, Oleg Granichin·· 13 小时前AI 评分31
Transformer 可精确编译为通用深度神经网络：去除激活-激活乘积的代价与多智能体注意力聚类
Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering
AI 导读
研究证明每个含 attention、LayerNorm 和 GELU MLP 的 Transformer 块都能精确编译为由固定仿射映射、路由及 square、sigmoid、逆平方根等算子构成的通用深度神经网络。
正文 · 原文
Abstract:Transformers are deployed as a special model class. Neural processing units (NPU) add dedicated operators for attention and normalization, and quantization relies on Transformer-specific algorithms. We show the computation does not force this. Every Transformer block, with attention, LayerNorm and a GELU MLP, compiles exactly into a generic deep neural network built from fixed affine maps, routing and the square, sigmoid and inverse square root, plus GELU or its sigmoid approximation. Without activation-activation products, every transform-wise compilation adds at least $N^2$ scalar nonlinear evaluations per head for dense scores. Keeping that one operation type removes this cost; its elementwise form, present on every surveyed NPU, suffices with the sigmoid and inverse square root. For top-$k$ attention with exchangeable tail scores, we prove that exact or conditional-mean tail mass gives an unbiased, minimum-mean-square-error estimate of the attention distribution, while tail drop is conditionally biased. When queries or agents cluster, centroids choose the keys first, reducing the number of computed scores from $N^2$ to $Nk$ per head. On DeiT-Tiny, MambaIRv2-Light and Pythia-70M the compiled models match their references with all keys, standard post-training quantization applies, and the exact tail-mass estimate beats tail dropping with half as many keys on DeiT-Tiny and, for $k\le8$, on Pythia-70M.
| Comments: | 36 pages, 6 tables and 2 figures in the main text, 8 supplementary tables. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2610.28958 [cs.LG] |
| (or arXiv:2610.28958v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28958 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sergey Salishev [view email]
[v1]
Tue, 20 Oct 2026 22:16:57 UTC (48 KB)
来源：arXiv cs.MA · arxiv.org