arXiv cs.AI· Sijie Xu, Francisco Cruz·· 10 小时前AI 评分18
结合动作前缀预测与跨模态对齐的多模态强化学习表示方法 IPR
Integrating Action Prefix Prediction and Alignment Across Modalities for Multimodal Reinforcement Learning
AI 导读
研究者提出 Improved Predictive Representation（IPR），用动作前缀预测结合基于 Contrastive Multimodal 学习的观测锚点对齐，替代依赖递归 rollout 的多模态联合预测表示。
正文 · 原文
Abstract:Learning representations from multimodal observations is difficult in reinforcement learning with limited interaction data. Multimodal Joint Predictive Representation combines multimodal latent prediction with contrastive learning across modalities, but its recursive rollout depends on earlier predictions and its pairwise alignment focuses on correspondence between modalities. We propose Improved Predictive Representation, which combines prediction from action prefixes with alignment at the observed anchor based on Contrastive Multimodal learning. At each horizon, it predicts future representations directly from the observed anchor and the corresponding sequence of actions. We evaluate two crop configurations: IPR-S with a minimum retained crop area of 10%, and IPR-H with 50%. Across five DeepMind Control Suite tasks, IPR-H achieves an average return across tasks that is 10.0% higher than the baseline with visual and depth images and 3.4% higher with visual images and position vectors. The relative performance of the two configurations varies across tasks. In Webots, both IPR configurations achieve higher mean final returns than the baseline without LiDAR, but not when LiDAR is available.
| Comments: | Accepted at the Australasian Conference on Robotics and Automation (ACRA 2026) |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13782 [cs.AI] |
| (or arXiv:2611.13782v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13782 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sijie Xu [view email]
[v1]
Tue, 10 Nov 2026 09:20:04 UTC (2,251 KB)
来源：arXiv cs.AI · arxiv.org