arXiv cs.AI· Naman Saxena, Vaneet Aggarwal·· 1 天前AI 评分18
约束双层强化学习：通过近似下层响应实现
Constrained Bilevel Reinforcement Learning via Approximate Lower-Level Responses
AI 导读
针对下层为约束马尔可夫决策过程、解映射可能多值且非 Lipschitz 的双层强化学习问题，提出将上层目标定义为固定随机下层算法返回策略的期望损失（近似响应），从而无需下层解唯一性。
正文 · 原文
Abstract:We study constrained bilevel reinforcement learning in which the lower-level problem is a constrained Markov decision process with a generally parameterized policy. The lower-level solution map can be set-valued and need not be Lipschitz continuous in the upper-level variable, so standard hypergradient analyses do not apply. We therefore define the upper-level objective as the expected upper-level loss of the policy returned by a fixed stochastic lower-level algorithm, which we call the approximate response. This objective is well defined without uniqueness of the lower-level solution. To compute its gradient, we randomize the reward and constraint values used during lower-level training without changing their means, so that the upper-level variable affects training only through known probabilities. This yields an unbiased estimator of the upper-level objective's gradient that does not differentiate through the lower-level updates. We establish lower-level reward and feasibility guarantees, upper-level stationarity, and a total sample complexity of $\widetilde{\mathcal{O}}(\delta_\ell^{-6}\epsilon_o^{-2})$ over both levels, where $\delta_\ell$ is the lower-level accuracy and $\epsilon_o$ is the upper-level stationarity error. Setting $\delta_\ell=\epsilon_o=\epsilon$ gives $\widetilde{\mathcal{O}}(\epsilon^{-8})$.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.00282 [cs.LG] |
| (or arXiv:2602.00282v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.00282 arXiv-issued DOI via DataCite |
Submission history
From: Naman Saxena [view email]
[v1]
Fri, 30 Jan 2026 20:10:21 UTC (51 KB)
[v2]
Thu, 12 Nov 2026 02:54:10 UTC (83 KB)
来源：arXiv cs.AI · arxiv.org