arXiv cs.AI· Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang, Bing Zheng, Zhaolin Wang, Yuanwei Liu·· 12 小时前AI 评分29
Martingale GRPO：为 VLA 强化学习按错误区间截断失败反馈
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
AI 导读
针对 GRPO 将单条轨迹级优势分配给所有动作、导致恢复尝试被一并惩罚的问题，研究者提出 MART-GRPO，用微调后的 VLM 在完整 rollout 视频中定位最后一个未被恢复的错误，仅保留该错误区间内的失败反馈。在 RoboTwin 2.0 上以 OpenVLA-OFT 实验，MART-GRPO 在多项任务上成功率超过全轨迹 GRPO。
正文 · 原文
Abstract:A failed vision-language-action (VLA) rollout does not imply that every action contributed to failure. Yet in group-relative policy optimization (GRPO), a single trajectory-level advantage is assigned to all actions in a rollout. Potentially useful recovery attempts can thus be penalized alongside the errors they aim to correct. Ideal stopping conditions for failure feedback are characterized using success-probability and policy-score martingales. At these stops, the removed gradient contribution has zero conditional expectation given the retained information. Truncation therefore preserves the expectation of a reference GRPO group-gradient estimator and can reduce its covariance. One such stop occurs when the conditional probability of success is zero, motivating error and recovery cues for selecting candidate feedback boundaries. Martingale GRPO (MART-GRPO) is introduced with a vision-language model (VLM) for event-guided truncation. The VLM is fine-tuned on our annotated training set using low-rank adaptation (LoRA). The last visible error without observed recovery is localized by the frozen VLM in complete rollout videos. Failure feedback is retained through the selected error interval and removed afterward, while successful-rollout loss terms and original loss normalization are preserved. For retrospective boundaries, changes in expectation and variance are analyzed to assess departures from ideal stopping. Experiments with OpenVLA-OFT on RoboTwin 2.0 show that MART-GRPO improves success rates over full-trajectory GRPO across multiple tasks.
| Comments: | 31 pages, 9 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32431 [cs.AI] |
| (or arXiv:2610.32431v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32431 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zongyao Zhao [view email]
[v1]
Fri, 23 Oct 2026 09:56:27 UTC (5,253 KB)
来源：arXiv cs.AI · arxiv.org