arXiv cs.AI· Mridul Mahajan, Aldo Pacchiano, Xuezhou Zhang·· 23 小时前AI 评分29
学习撤销:状态空间变换下的强化学习迁移
Learning to Undo: Transfer Reinforcement Learning under State Space Transformations
AI 导读
研究者提出一种强化学习迁移算法,通过学习一个“撤销映射”将目标 MDP 的状态空间还原为源 MDP,从而把源策略迁移到目标环境。理论分析表明,在撤销映射为线性且源满足线性-$Q^\star$ 可实现时,该方法样本复杂度严格优于在目标 MDP 上从头训练。在连续控制任务和 Atari 游戏上,该方法显著提升了样本效率。
正文 · 原文
Abstract:Transfer learning in reinforcement learning (RL) has shown strong empirical success. In this work, we take a more principled perspective by studying when and how transferring knowledge between MDPs (a source and a target) can be provably beneficial. Specifically, we consider the case where there exists an undo map such that applying this map to the target's state space recovers the source exactly. We propose an algorithm that learns this map by matching state feature statistics gathered from both MDPs, and then uses it to transfer the source policy to the target. We theoretically justify the algorithm by analyzing the setting when the undo map is linear and the source is linearly-$Q^\star$ realizable, where our approach has strictly better sample complexity than tabula rasa RL in the target MDP. Empirically, we demonstrate that these benefits extend beyond this regime: on challenging continuous control tasks and Atari games, our method achieves significantly better sample efficiency. Overall, our results highlight how shared structure between tasks can be leveraged for efficient transfer of policies across environments.
| Comments: | NeurIPS'26 paper |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23081 [cs.LG] |
| (or arXiv:2611.23081v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23081 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mridul Mahajan [view email]
[v1]
Tue, 17 Nov 2026 17:06:51 UTC (2,471 KB)
来源:arXiv cs.AI · arxiv.org