arXiv cs.AI· Yuhan Wang, Ruiyang Wang, Jiaguan Zhu·· 23 小时前AI 评分31
Outcome-Balanced PPO 与策略诱导流矫正:面向流式 VLA 策略的强化学习框架
Outcome-Balanced PPO and Policy-Induced Flow Rectification for Flow-Based VLA Policies
AI 导读
研究提出一种结合 Outcome-Balanced PPO 与策略诱导流矫正的强化学习框架,用于流式 VLA 策略的后训练。Outcome-Balanced PPO 利用任务成功率估计对正优势重加权,并在保持符号的前提下缩放正负优势;流矫正则周期性对当前策略生成的 ODE 端点对拟合线性桥流。在模拟操作基准上,该框架成功率优于所评估的基线,并在 Real2Sim2Real 任务中考察了组件组合。
正文 · 原文
Abstract:Reinforcement learning (RL) enables post-training of flow-based vision-language-action (VLA) policies by optimizing expected task return through environment interaction. However, this optimization raises two challenges: policy updates can alter the velocity-field geometry underlying the score approximation used for stochastic transitions, and sparse multi-task rollouts can yield uneven learning signals across tasks and advantage signs. We introduce an RL framework that combines Outcome-Balanced Proximal Policy Optimization (PPO) with policy-induced flow rectification. Outcome-Balanced PPO reweights positive advantages using task success-rate estimates and rescales positive and negative advantages while preserving their signs. Policy-induced rectification periodically fits a linear-bridge flow to ODE endpoint pairs generated by the current policy, refreshing the representation used in subsequent stochastic optimization. We evaluate the framework on simulated manipulation benchmarks and additionally study a component combination in a Real2Sim2Real task, examining task performance, optimization behavior, and flow consistency. The proposed framework improves success rates over the evaluated baselines across the simulated benchmarks.
| Comments: | 15 pages |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.22878 [cs.AI] |
| (or arXiv:2611.22878v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22878 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhan Wang [view email]
[v1]
Tue, 17 Nov 2026 14:52:57 UTC (820 KB)
来源:arXiv cs.AI · arxiv.org