arXiv cs.CL· Linke Zhang·· 13 小时前AI 评分20
baseVultra：以容量重分配实现紧凑非线性循环网络
baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity
AI 导读
baseVultra 是一种紧凑循环单元，将稠密循环预算集中到单个 H×H 候选生成矩阵，并用 O(H) 对角交互替代循环门矩阵。在 TinyShakespeare 单层 200K 设置下验证损失为 1.594，优于参数匹配的 LSTM（1.604）和 GRU（1.616），稠密循环矩阵向量计算量约为同隐藏维度 GRU 的 54%。
正文 · 原文
Abstract:Nonlinear recurrent networks remain useful for streaming sequence modeling, yet gated cell design is still dominated by LSTM and GRU. We study whether recurrent computation can be reallocated away from gate transforms and toward candidate generation. We propose baseVultra, a compact recurrent cell that concentrates the dense recurrent budget on a single H x H candidate-generation matrix, while replacing recurrent gate matrices with O(H) diagonal interactions. The candidate update combines this dense branch with two low-rank auxiliary branches, an EMA cache, and learnable output blending; the gates additionally receive lightweight second-order access to h_{t-2} through diagonal terms. On TinyShakespeare character-level language modeling, baseVultra achieves a validation loss of 1.594 in the compact one-layer 200K setting, compared with 1.604 for a parameter-matched LSTM and 1.616 for a parameter-matched GRU, while its dense recurrent matrix-vector computation is about 54% of GRU under matched hidden size for this configuration. On text8, a roughly 45x larger benchmark, baseVultra is competitive with full-gate cells (1.854 bpc vs. 1.849 for GRU and 1.837 for LSTM). On formal-language probes, the largest gains appear under constrained hidden-state capacity (0.991 vs. 0.761 GRU and 0.691 LSTM OOD2 accuracy on Counter at H=16) and vanish as hidden size grows. These results characterize where and when candidate-focused recurrent capacity allocation helps: the benefit is concentrated in compact regimes, while full-gate cells remain competitive at scale.
| Comments: | Preprint. 14 pages, 3 figures. Revised based on peer review feedback from WAIC Academic 2026 (1 weak accept, 2 borderline) |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.30154 [cs.CL] |
| (or arXiv:2610.30154v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30154 arXiv-issued DOI via DataCite |
Submission history
From: Linke Zhang [view email]
[v1]
Sun, 23 Aug 2026 14:31:35 UTC (46 KB)
来源：arXiv cs.CL · arxiv.org