arXiv cs.AI· Pengzhi Yang, Xinyu Wang, Pengyu Jing, Kehan Wen, Yiduo Qu, Zhenhao Huang, Minghao Fu, Xin Liu, Yaheng Shen, Fan Shi·· 3 天前AI 评分22
RARM：面向机器人操作的置信门控进度奖励建模
RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation
AI 导读
研究者提出 Reference-Anchored Reward Model（RARM），一种轻量视觉比较器，可将单条成功演示转化为稠密的进度感知奖励，训练仅用通用视频和对比式时序目标，无需机器人数据或任务专属奖励标注。
正文 · 原文
Abstract:Reinforcement learning for robot manipulation is often bottlenecked by reward design, especially in long-horizon tasks: sparse success rewards provide weak supervision, while hand-crafted dense rewards are tedious to design and generalize poorly across tasks. Progress-based reward models offer a promising alternative by estimating how far an observation has advanced toward task completion, but existing approaches often require task-specific demonstrations or progress labels, and can assign high rewards to visually plausible but physically incorrect states. We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward. RARM is trained once on general-purpose videos with a contrastive temporal objective, requiring no robot-specific data, task-specific reward labels, or per-task reward engineering. At deployment, RARM matches rollout clips to reference clips and rewards only confident forward progress, suppressing uncertain matches that may otherwise produce false-positive rewards. Across 9 simulated manipulation tasks from LIBERO and MetaWorld and 4 real-world tasks, RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding, where unreliable progress estimates are especially harmful.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.22027 [cs.RO] |
| (or arXiv:2606.22027v5 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2606.22027 arXiv-issued DOI via DataCite |
Submission history
From: Pengzhi Yang [view email]
[v1]
Sat, 20 Jun 2026 13:03:21 UTC (16,178 KB)
[v2]
Wed, 24 Jun 2026 05:16:35 UTC (16,178 KB)
[v3]
Mon, 24 Aug 2026 04:09:24 UTC (16,181 KB)
[v4]
Tue, 25 Aug 2026 07:44:37 UTC (16,181 KB)
[v5]
Mon, 12 Oct 2026 07:47:18 UTC (13,680 KB)
来源：arXiv cs.AI · arxiv.org