arXiv cs.AI· Armand Foucault (IMT), Franck Mamalet (Comue de Toulouse), Fran\c{c}ois Malgouyres (IMT)·· 12 小时前AI 评分18
正交循环神经网络量化研究：QORNN 的两种学习策略与全量化模型表现
On Orthogonal Recurrent Neural Networks Quantization
AI 导读
研究提出 QORNN，通过量化感知训练（QAT）结合正交投影的两种策略学习量化近似正交 RNN，并研究激活值的训练后量化以实现循环环路的纯整数计算。最高效的全量化模型是目前唯一能求解 1000 时间步 Copy-task 的方案，在 permuted MNIST 及部分 IoT 任务等长程依赖任务上超越现有量化模型，但在 LSTM 等门控 RNN 更擅长的任务上表现较弱。
正文 · 原文
Abstract:In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the Copy-task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices. In this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient fully-quantized models are currently the only alternatives for solving the Copy-task with 1000 timesteps. They also surpass existing quantized models on tasks requiring long-term dependencies, such as permuted MNIST as well as some Internet of Things (IoT) tasks. However, they are less effective on tasks where long short-term memory (LSTMs) and other gated RNNs outperform ORNNs.
| Comments: | Progress in Artificial Intelligence, In press |
| Subjects: | Neural and Evolutionary Computing (cs.NE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Signal Processing (eess.SP); Statistics Theory (math.ST) |
| Cite as: | arXiv:2402.04012 [cs.NE] |
| (or arXiv:2402.04012v3 [cs.NE] for this version) | |
| https://doi.org/10.48550/arXiv.2402.04012 arXiv-issued DOI via DataCite |
Submission history
From: Francois Malgouyres [view email] [via CCSD proxy]
[v1]
Mon, 5 Feb 2024 09:59:57 UTC (3,509 KB)
[v2]
Mon, 10 Jun 2024 11:40:40 UTC (2,492 KB)
[v3]
Fri, 23 Oct 2026 16:19:05 UTC (420 KB)
来源：arXiv cs.AI · arxiv.org