arXiv cs.AI· Leo Dobson, Anton Ragni·· 10 小时前AI 评分31
RNN 的时序秩瓶颈:结构化高维状态动力学如何恢复时序容量
Temporal Rank in Recurrent Neural Networks
AI 导读
研究指出 RNN 存在一个独立于梯度消失/爆炸的固有限制:秩受限的隐藏状态动力学造成信息瓶颈,压缩了可长期保留的独立输入方向数量,即输入到状态映射的秩。作者提出结构化高维状态动力学,在保持时序可分性的同时防止输入到状态 Jacobian 随时间饱和,模型仍严格属于标准 RNN 形式,在受控序列任务上学习速度显著加快。结果表明 RNN 性能由时序秩的演化与保持决定,仅缓解梯度问题不足以维持时序容量。
正文 · 原文
Abstract:Recurrent Neural Networks (RNNs) are a natural framework for temporal modelling, yet are often dismissed due to their sequential, non-parallelisable computation and difficulty in capturing long-range dependencies, typically attributed to vanishing and exploding gradients. We identify a distinct and under-explored limitation: an information bottleneck induced by rank-constrained hidden state dynamics, which limits the number of independent input directions that can be preserved over time, as reflected in the rank of the input-to-state mapping. We show that this constraint restricts temporal capacity independently of gradient stability. To address this, we introduce structured high-dimensional state dynamics that preserve temporal separability and prevent saturation of the input-to-state Jacobian over time. On controlled sequence tasks, the resulting model exhibits rapid learning despite remaining strictly within a standard RNN formulation. Our results demonstrate that RNN performance is governed by the evolution and preservation of temporal rank, and suggest that mitigating gradient pathologies alone is insufficient without explicitly maintaining temporal capacity.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24394 [cs.LG] |
| (or arXiv:2611.24394v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24394 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Anton Ragni [view email]
[v1]
Wed, 18 Nov 2026 13:28:07 UTC (3,301 KB)
来源:arXiv cs.AI · arxiv.org