arXiv cs.AI· Likui Zhang, Zhe Zhu, Zhihao Zhan, Yibo Bai, Hanlin Wang, Dongwei Li, Bin Wang, Tianle Zhang, Xiaodan Liang, Liang Lin·· 13 小时前AI 评分31
AtomicRL：面向长程机器人操作的原子强化学习框架
AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
AI 导读
AtomicRL 是一个面向 VLA 模型的强化学习后训练框架，通过原子任务结构实现可靠的信度分配与自适应探索。在 LIBERO-LONG 上，它相对 PPO 在两种 VLA 架构上分别取得 3.6% 和 4.0% 的绝对性能提升；在 CALVIN ABC-D 上全任务性能最高提升 9.5%。配合 Real2Sim2Real 流程，真实机器人任务性能平均提升 21.3%。
正文 · 原文
Abstract:Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.30097 [cs.RO] |
| (or arXiv:2610.30097v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30097 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Likui Zhang [view email]
[v1]
Wed, 21 Oct 2026 17:45:24 UTC (5,593 KB)
来源：arXiv cs.AI · arxiv.org