arXiv cs.AI· Zekang Wang, Zhe He, Gaojie Jin·· 6 天前AI 评分29
部分可观测 RL 中学习循环网络是否必要？冻结循环网络可作为强基线
Does Learning Recurrent Networks Matter in Partially Observable RL?
AI 导读
一项研究探讨了部分可观测强化学习中是否必须训练循环网络，提出将固定循环网络与可训练非线性输出模块结合的简单架构。在多种记忆需求、观测模态和动作空间的任务上，冻结循环网络可支持数百步的回溯，并取得有竞争力的控制性能，同时可训练参数量不到全可训练版本的一半。结果表明，学习循环记忆模块并非解决 POMDP 的必要条件，冻结循环网络为部分可观测 RL 提供了参数高效且表现强劲的基线，源代码已公开。
正文 · 原文
Abstract:Reinforcement learning (RL) under partial observability typically re- lies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving par- tially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple archi- tecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and sur- prisingly strong baseline for partially observable RL. Our source code is available at this https URL
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.13197 [cs.LG] |
| (or arXiv:2610.13197v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13197 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zekang Wang [view email]
[v1]
Fri, 9 Oct 2026 08:17:28 UTC (1,326 KB)
来源：arXiv cs.AI · arxiv.org