arXiv cs.AI· Zeyu Fan, Alexandru-Dragos Manolache, Jan van Gemert·· 3 天前AI 评分22
RNN Homunculus：如何学习门控长期记忆
The RNN Homunculus: Learning to Gate Long-Term Memory
AI 导读
研究通过受控序列分类任务发现，门控 RNN 的瓶颈不在加法复制路径，而在门值本身需要另一个 RNN 学习，当所有训练样本都依赖长程记忆时同样受困于梯度消失。加入短依赖样本可让门控模型学到能泛化到长依赖样本的解，训练得当的门控 RNN 测试准确率可匹配甚至超过 Mamba 和 xLSTM。
正文 · 原文
Abstract:Learning long-range dependencies with recurrent neural networks (RNNs) is difficult because of the vanishing gradient problem. Gated recurrent neural networks like GRUs and LSTMs address this problem and are commonly explained by their ability to create additive copy paths through time. This explanation is correct, but incomplete: useful RNN gate values must themselves be learned, which is typically done with another RNN. We study this gate-learning problem through carefully controlled sequence classification tasks. Our analysis shows that gated architectures struggle when all training samples require long-range memory and thus equally suffer from the vanishing gradient problem. We intervene by adding short-dependency samples which let gated models learn solutions that capture long-dependency samples, but also generalize across different class labels. When properly learned, gated RNNs match or even exceed the test accuracy of modern architectures like Mamba and xLSTM. Overall, our results highlight gate learning as a central part of learning long-range memory and show how training data can support this process by exposing the same task over shorter dependencies.
| Comments: | 20 pages, including references and appendices |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14279 [cs.LG] |
| (or arXiv:2610.14279v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14279 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zeyu Fan [view email]
[v1]
Fri, 9 Oct 2026 19:59:35 UTC (2,999 KB)
来源：arXiv cs.AI · arxiv.org