arXiv cs.AI· Pekka Malo, Lauri Viitasaari, Antti Suominen, Eeva Vilkkumaa, Olli Tahvonen·· 13 小时前AI 评分18
失败风险的价格：带凸正则化的截断屏障策略梯度
The Price of Failure Risk: Clipped Barrier Policy Gradients with Convex Regularization
AI 导读
研究提出一种预算约束下的策略梯度方法：将剩余预算放入状态，减去随预算收缩而增长的屏障项，并把组合奖励从下方截断在 -m，m 即“失败风险的价格”。理论证明在沿网络单元运动凸的参数正则化下，种群训练流呈指数收敛，把策略梯度的平均场收敛从相对熵扩展到权重衰减。实验在维护基准、油箱控制器等任务上测量了界的松弛度。
正文 · 原文
Abstract:Many operations run on a budget that must rarely run out, such as a wear allowance. A wrapper keeps the remaining budget in the state, subtracts a barrier that grows as the budget shrinks and clips the combined reward from below at -m, and a standard policy-gradient learner trains unchanged. The clip level m is a price of failure risk, since the trained value splits exactly into utility before failure, the cost of a margin and the price times the risk, so whatever policy training returns earns the most such utility net of that cost among policies with no greater risk, up to its optimization error. Its risk exceeds a comparison policy's by at most order 1/m when charged costs stay bounded, and value or validation data certify it. The main theoretical result is exponential convergence of the population training flow under any parameter regularizer convex along the motions of the network's units and strong enough against the price, biased toward the prior policy. This extends mean-field convergence of policy gradients from relative entropy to weight decay, which covers continuous-time training of finite networks. Relative entropy adds noise that lets learning leave flat regions, weight decay pulls the parameters toward a reference, and action regularization bounds the risk change by the distance to a trusted rule. Experiments on an exact maintenance benchmark, a fuel-tank controller, a solvable flow and a benchmark-suite task measure the bounds' slack. We state when another method is better, without claiming this one outperforms them.
| Comments: | 38 pages, 7 figures, 2 tables; Online Resource 1 (44 pages) as an ancillary file. Revised and corrected version, replaces v2 (title changed). Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Probability (math.PR); Machine Learning (stat.ML) |
| MSC classes: | 90C40, 93E20, 68T05, 49Q22, 90C15 |
| ACM classes: | I.2.6; G.1.6 |
| Cite as: | arXiv:2411.19193 [cs.LG] |
| (or arXiv:2411.19193v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2411.19193 arXiv-issued DOI via DataCite |
Submission history
From: Pekka Malo [view email]
[v1]
Thu, 28 Nov 2024 15:04:43 UTC (104 KB)
[v2]
Tue, 16 Sep 2025 14:10:15 UTC (53 KB)
[v3]
Tue, 27 Oct 2026 14:54:27 UTC (1,317 KB)
来源：arXiv cs.AI · arxiv.org