arXiv cs.AI· Julian Sester, C\'ecile Decker·· 12 小时前AI 评分20
Q-Learning 在有限模型不确定性下的鲁棒算法
Q-Learning under Finite Model Uncertainty
AI 导读
针对每个状态-动作对仅关联有限候选转移核集合的马尔可夫决策过程，提出一种鲁棒 Q-learning 算法，学习器只需从各候选模拟器观测后继状态及其确定性奖励，无需已知转移概率或奖励函数。
正文 · 原文
Abstract:We propose a robust Q-learning algorithm for Markov decision processes under model uncertainty when each state--action pair is associated with a finite ambiguity set of candidate transition kernels. This finite-measure framework enables user-designed uncertainty models, including competing fitted models and stress scenarios. {The learner observes successor states and their deterministic rewards from every candidate simulator, without requiring explicit knowledge of the transition probabilities or the reward function.} We establish almost sure convergence of the learned Q-function to the robust optimum, and derive a time-uniform high-probability error certificate that separates an observable empirical Bellman residual from transition-kernel estimation error. We also give quantitative finite-ambiguity approximation results for Wasserstein and parametric uncertainty models on finite state spaces. Numerical examples illustrate the effect of scenario selection and compare the tabular algorithm with robust value iteration.
| Subjects: | Optimization and Control (math.OC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Probability (math.PR) |
| Cite as: | arXiv:2407.04259 [math.OC] |
| (or arXiv:2407.04259v4 [math.OC] for this version) | |
| https://doi.org/10.48550/arXiv.2407.04259 arXiv-issued DOI via DataCite |
Submission history
From: Julian Sester [view email]
[v1]
Fri, 5 Jul 2024 05:19:36 UTC (49 KB)
[v2]
Sun, 16 Feb 2025 03:16:16 UTC (53 KB)
[v3]
Mon, 9 Feb 2026 05:38:32 UTC (348 KB)
[v4]
Thu, 5 Nov 2026 07:58:02 UTC (245 KB)
来源：arXiv cs.AI · arxiv.org