arXiv cs.AI· Yuchen Jiao, Gen Li·· 15 小时前AI 评分18
面向鲁棒强化学习的可证明高效 Q-learning 算法
Provably Efficient Q-learning for Robust Reinforcement Learning
AI 导读
研究提出一种面向鲁棒强化学习的异步 Q-learning 算法，在基于总变差（TV）距离构建不确定性集的无限时域 RMDP 上，仅用单条马尔可夫样本轨迹即可实现最优 Q-function 的逐元素 ε-近似。其样本复杂度上界接近 minimax 最优，仅多出 1/(1-γ) 因子，并显著优于现有结果。
正文 · 原文
Abstract:Standard reinforcement learning (RL) algorithms based on Markov Decision Processes (MDPs) are vulnerable to sim-to-real gaps. To address this issue, distributionally robust RL algorithms based on robust MDPs (RMDPs) have been proposed, for ensuring that agents can perform reliably to unpredictable conditions in real-world environments. Despite of the rich literature, \revision{an efficient model-free method is still lacking} for robust RL, particularly for Markovian data, which is more relevant in practical applications. In this work, we take a step towards settling this problem by proposing a sample efficient (asynchronous Q-learning) algorithm for robust RL. In particular, consider a $\gamma$-discounted infinite-horizon RMDP with state space $\mathcal{S}$ and action space $\mathcal{A}$, where the uncertainty set is specified by the total variation (TV) distance with uncertainty level $\sigma$. Assuming access to a single trajectory of Markovian samples induced by a behavior policy and the nominal transition kernel, the sample complexity of the proposed Q-learning algorithm --- to achieve an entrywise $\varepsilon$-approximation of the optimal Q-function --- is bounded by $$ \frac{1}{\mu_{\mathsf{min}}\varepsilon^2(1-\gamma)^3\max\{1-\gamma,\sigma\}} + \frac{t_{\mathsf{mix}}}{\mu_{\mathsf{min}}(1-\gamma)} $$ up to logarithmic factors. Here, $t_{\mathsf{mix}}$ and $\mu_{\mathsf{min}}$ denote the mixing time and the minimum state-action occupancy probability of the sample trajectory, respectively. This result is nearly minimax optimal, with an additional factor of $\frac{1}{1-\gamma}$. Moreover, it achieves a significant improvement over existing bounds.
| Subjects: | Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Statistics Theory (math.ST) |
| Cite as: | arXiv:2610.35656 [stat.ML] |
| (or arXiv:2610.35656v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35656 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuchen Jiao [view email]
[v1]
Mon, 26 Oct 2026 16:03:25 UTC (170 KB)
来源：arXiv cs.AI · arxiv.org