arXiv cs.AI· Zhen Gao, Xiao Yin, Kexin Liang, Pedro Reviriego, Shanshan Liu, Fabrizio Lombardi·· 10 小时前AI 评分18
二进制 Transformer 搭配近似内存:PIES 置换不变性选择性保护方案
Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection
AI 导读
研究以 BERT 为例评估二进制 Transformer 在近似内存下的可靠性,发现仅在 BER 达 5% 或 10% 时才出现大幅精度损失。为此提出 PIES 方案,利用 Transformer 对行列置换的不变性分组关键权重并做选择性保护,在 BER 高达 10% 时以远低于传统纠错方案的额外开销恢复大部分性能损失。PIES 代码已在 GitHub 开源。
正文 · 原文
Abstract:Quantization is an effective technique to reduce memory storage, computation complexity and power for Transformers, and the binarization of the weights is important for meeting these requirements. Approximate memory can further reduce the power for memory, but it also introduces errors. Current research for binary Transformers focuses on task performance, but the model dependable operation against errors on the memory has not been well studied. In this paper, we first evaluate the dependability of binary Transformers with the Bidirectional Encoder Representations from Transformers (BERT) as a case study. A large accuracy loss occurs only at high Bit Error Rates (e.g. BER = 5% or 10%). Then we propose a novel Permutation Invariance Enabled Selective (PIES) scheme to protect the binary weights at very low overhead. This scheme exploits the invariance of Transformers to permutations in the rows and columns to group the critical weights and perform selective protection on those weights. Simulation results show that PIES can recover most of the performance loss caused by bit errors at BERs as high as 10%, at an overhead significantly lower than traditional error correction schemes. These results confirm the feasibility of running binary Transformers using approximate memory with aggressive voltage scaling on resource-limited low power edge devices. The code for PIES is released on github (this https URL).
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23358 [cs.LG] |
| (or arXiv:2611.23358v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23358 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shanshan Liu [view email]
[v1]
Tue, 17 Nov 2026 15:06:16 UTC (4,659 KB)
来源:arXiv cs.AI · arxiv.org