arXiv cs.AI· Joonyoung Lim, Younghwan Yoo·· 12 小时前AI 评分29
用大规模并行加速安全强化学习
Accelerating Safe Reinforcement Learning with Massive Parallelism
AI 导读
针对大规模并行训练中短 rollout 与 CMDP 全回合成本约束不匹配的问题，研究者将错峰环境重置建模为短轨迹片段上的相位混合分布，并提出相位聚合成本估计器，从短 rollout 批次重建回合级成本估计。
正文 · 原文
Abstract:We investigate the problem of scaling safe reinforcement learning (safe RL) to massively parallel training regimes, where a large number of environments are exe- cuted with short rollout horizons. Although this setting improves data throughput and wall-clock efficiency, it creates a mismatch with constrained Markov decision processes (CMDPs), whose safety constraints are defined over full-episode cost returns. We address this mismatch by analyzing staggered environment resets as a phase-mixture distribution over short trajectory segments. This analysis charac- terizes the bias induced by phase coverage and motivates a phase-aggregated cost estimator that reconstructs episode-level cost estimates from short rollout batches. We further incorporate finite-sample uncertainty and distribution mismatch into a constraint-tightening framework, yielding a scalable Safe RL method compatible with on-policy optimization. Using an MJX-based implementation of Safety Gym, our experiments show that the proposed method achieves performance comparable to non-massively parallel implementations while substantially reducing wall-clock training time.
| Comments: | Accepted at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.08273 [cs.LG] |
| (or arXiv:2611.08273v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08273 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joonyoung Lim [view email]
[v1]
Thu, 5 Nov 2026 16:43:57 UTC (1,450 KB)
来源：arXiv cs.AI · arxiv.org