arXiv cs.AI· Nikolai Smolyanskiy, Jonathan Shock·· 9 天前AI 评分22
RSSM 世界模型中的奖励可观测性与离线检查点选择的局限
Reward Observability and the Limits of Offline Checkpoint Selection in RSSM World Models
AI 导读
研究在 LunarLander-v3 上训练的 RSSM 世界模型，其基于世界模型检查点 280 的 A2C 策略在 100 个留出回合上平均回报 +189.5，追平最佳无模型 A2C 检查点（+183.7），但真实训练转移少约 65 倍。作者提出奖励可观测分数（ROF），发现其数值随潜变量缩放变化、跨模型不可比，且无法预测 MPC 的闭环崩溃；筛查的 108 个离线指标均未通过置换检验。
正文 · 原文
Abstract:We study the closed-loop properties of a recurrent state-space model (RSSM) world model trained on human demonstrations in Gymnasium's LunarLander-v3. We use the trained world model for zero-shot CEM model-predictive control (MPC) and for actor-critic (A2C) training in imagination. Scored on 100 held-out episodes, the selected model-based A2C policy (trained on world-model checkpoint 280) reaches a mean return of +189.5, matching the best model-free A2C checkpoint (+183.7; 600- and 1000-step episode caps respectively) with ~65x fewer real training transitions. We also compare world-model MPC with a behaviour-cloning (BC) policy trained on the successful demonstrations. The BC policy matches MPC's mean return only under stochastic action selection, and on the same 20 episodes it has one catastrophic episode where MPC has none. We then introduce the Reward Observability Fraction (ROF), the Euclidean fraction of the reward gradient in the observable subspace of the linearized latent dynamics, and show that the next H observations carry Fisher information about every direction in this subspace and none about directions orthogonal to it. ROF itself however depends on how the latent is scaled: rescaling it changes ROF but not the model, so raw levels are not comparable across models. Forcing the reward head onto the posterior-corrected latent z raises ROF, and the rise survives a coordinate-invariant check on the pair of runs we tested. Finally, we test whether ROF or other offline metrics can predict the closed-loop collapse of MPC. Collapse varies between training runs with identical data and configuration. ROF does not predict it, none of the 108 offline summaries we screened passes a permutation test, and the best candidate fails on new runs. Predicting collapse offline from the model and logged data alone remains open.
| Comments: | Preprint, 22 pages (17 main text + 5 pages appendix), 4 figures, 9 tables. Video: this https URL , Code: this https URL and this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Systems and Control (eess.SY) |
| ACM classes: | I.2.6; I.2.8; I.2.9 |
| Cite as: | arXiv:2607.01736 [cs.LG] |
| (or arXiv:2607.01736v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01736 arXiv-issued DOI via DataCite |
Submission history
From: Nikolai Smolyanskiy [view email]
[v1]
Thu, 2 Jul 2026 05:46:03 UTC (2,597 KB)
[v2]
Sat, 4 Jul 2026 04:11:55 UTC (2,597 KB)
[v3]
Thu, 8 Oct 2026 02:14:26 UTC (2,534 KB)
来源：arXiv cs.AI · arxiv.org