arXiv cs.AI· Hanzhen Wang, Boxiang Lyu, Zhongxiang Dai, Siyang Gao, Dake Zhang, Shuang Qiu·· 2 天前AI 评分18
基于优化确定性等价的风险敏感离线强化学习：可证明高效的算法与上下界
Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents
AI 导读
研究用优化确定性等价（OCE）统一刻画风险敏感离线强化学习，提出动态-OCE 与静态-OCE 两种形式。在表格设定下给出悲观值迭代算法及紧的次线性次优上界，并建立 minimax 下界；针对大状态空间进一步提出悲观最小二乘值迭代算法，将 OCE 风险准则融入线性函数逼近，给出依赖效用函数与特征维度 d 的次线性保证。作者称这是动态与静态 OCE RL 的首个离线算法及首个上下界。
正文 · 原文
Abstract:Risk-sensitive reinforcement learning (RL) addresses decision-making under uncertainty and adverse outcomes. Learning risk-sensitive policies from fixed datasets is important when additional interaction with the environment is costly. Existing work on risk-sensitive RL focuses on online settings or specific risk measures, leaving a general theoretical understanding of its offline setting underdeveloped. In this work, we investigate efficient offline risk-sensitive RL using the optimized certainty equivalent (OCE), a general class of risk measures. We consider two formulations: dynamic-OCE RL, which applies OCE recursively at each step, and static-OCE RL, which applies a single OCE to the cumulative rewards. Under the tabular setting, we develop pessimistic value-iteration algorithms for both formulations and establish tight sublinear suboptimality bounds, explicitly capturing how risk sensitivity, encoded by utility functions, affects learning difficulty. Moreover, we establish minimax suboptimality lower bounds to characterize the problem's fundamental limits. To handle large state spaces, we further develop pessimistic least-squares value-iteration algorithms that incorporate the general OCE risk criterion into linear function approximation. We establish utility-dependent sublinear guarantees with explicit dependence on the feature dimension $d$. To our knowledge, we provide the first offline algorithms along with the first upper and lower suboptimality bounds for both dynamic- and static-OCE RL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.21870 [cs.LG] |
| (or arXiv:2610.21870v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21870 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanzhen Wang [view email]
[v1]
Thu, 15 Oct 2026 13:41:18 UTC (138 KB)
来源：arXiv cs.AI · arxiv.org