arXiv cs.AI· Chenxi Li, Yao Shu, Siyang Gao, Shuang Qiu, Guangxin Jiang·· 14 小时前AI 评分26
安全多目标强化学习：零约束违反与原始平均收敛
Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence
AI 导读
研究提出安全多目标强化学习算法，结合乐观奖励估计与悲观成本上界，使T个回合内执行的每个策略以高概率实现零约束违反，并在平滑Tchebycheff标量化下达到Õ(1/√T)的次优间隙。进一步给出无需预设偏好的算法，先安全探索再对任意偏好规划，保持相同安全保证并达到Õ(1/√T+1/√K)次优间隙。这是首批对每个执行策略都保证安全的样本高效MORL算法。
正文 · 原文
Abstract:Many applications of multi-objective reinforcement learning (MORL) require policies that balance conflicting objectives while satisfying safety constraints. We study safe MORL with zero constraint violation throughout learning, covering both cumulative and stepwise constraints. A key challenge is to explore sufficiently under unknown dynamics and constraint costs without violating the constraints. For a given preference over objectives, we develop an algorithm to learn a safe Pareto optimal policy by combining optimistic reward estimates with pessimistic cost bounds. With high probability, every policy executed over $T$ episodes incurs zero constraint violation. The algorithm returns a safe Markov policy induced by the averaged occupancy measure and achieves an $\widetilde{O}(1/\sqrt{T})$ suboptimality gap under smooth Tchebycheff scalarization. We further develop a preference-free algorithm that first explores safely without a prespecified preference and then computes a safe policy for any given preference without further interaction. Over $T$ exploration episodes and $K$ planning iterations, it retains the same safety guarantee and achieves an $\widetilde{O}(1/\sqrt{T}+1/\sqrt{K})$ suboptimality gap. Both algorithms require knowledge of a policy that strictly satisfies the constraints and its costs. We then relax these requirements, incurring at most constant cumulative violation. To our knowledge, we provide the first sample-efficient MORL algorithms that guarantee safety for every executed policy.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.39409 [cs.LG] |
| (or arXiv:2610.39409v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39409 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guangxin Jiang [view email]
[v1]
Thu, 29 Oct 2026 05:47:57 UTC (57 KB)
来源：arXiv cs.AI · arxiv.org