arXiv cs.AI· Mattie Fellows, Clarisse Wibault, Uljad Berdica, Johannes Forkel, Maike Osborne, Jakob N. Foerster·· 11 小时前AI 评分32
SOReL 与 TOReL：实现完全离线强化学习的方法框架
Fully Offline Reinforcement Learning
AI 导读
研究者提出 SOReL，一种完全离线的贝叶斯基于模型强化学习方法，通过学习动力学后验、用预测不确定性估计策略价值，实现全程离线的超参数选择。其扩展 TOReL 将该调参框架推广到任意无模型与基于模型的离线 RL 算法。作者给出 regret 分析，证明贝叶斯离线 RL 在标准正则条件下达到 minimax 最优参数速率。
正文 · 原文
Abstract:Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2505.22442 [cs.LG] |
| (or arXiv:2505.22442v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.22442 arXiv-issued DOI via DataCite |
Submission history
From: Mattie Fellows [view email]
[v1]
Wed, 28 May 2025 15:07:24 UTC (5,896 KB)
[v2]
Thu, 29 May 2025 20:38:03 UTC (5,896 KB)
[v3]
Wed, 15 Jul 2026 19:14:08 UTC (6,122 KB)
[v4]
Fri, 30 Oct 2026 17:26:53 UTC (6,122 KB)
来源：arXiv cs.AI · arxiv.org