arXiv cs.AI· Jiaxuan Luo, Xingguo Xu, Shanshan Wang, Yuhan Zhou, Zhen Zhang·· 10 小时前AI 评分44
CriticHack:评估机器人策略优化下的视觉奖励模型
CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
AI 导读
研究发现视觉奖励模型可通过成功/失败区分与任务成功率上升两项检查,却仍让策略学会操作错误物体:用 Robometer 微调扩散策略开抽屉时任务成功率升 10.2 个百分点,开错抽屉的失败率同时升 10.9 个百分点,RoboReward-8B 与真机实验重现该现象。
正文 · 原文
Abstract:Learned visual reward models score a robot's execution from camera images and an instruction, and are increasingly used to fine-tune robot policies. Two checks decide whether to trust one: the reward model separates successes from failures, and task success rises during training. We show that both checks can pass while the policy learns to act on the wrong object. When a diffusion policy is fine-tuned against the reward model Robometer to open a drawer, task success rises by 10.2 percentage points. Wrong-object failures, which open a different drawer, rise by 10.9 points, whereas the same training on the simulator's task-completion signal raises success alone. A second reward model, RoboReward-8B, behaves the same way, and the effect reappears on a physical robot. This happens because of how the reward scores the policy's own failures. Most failures make little progress and score low, while a wrong-object failure looks like progress and scores well above them. Optimization shifts a policy toward outcomes that score above the executions they compete with, the executions they can take the place of. It therefore turns low-scoring failures into both successes and wrong-object failures, even when the reward model scores successes higher. This rule needs only executions of the initial policy, so it can be tested before training. On a stacking task we predicted which of three object placements Robometer would exploit, and training raised wrong-object stacks from 20% to 69% in that placement and in neither of the others. The same rule gives a check to run before optimization and a repair that needs no labels: scoring each execution again on frames re-rendered with the objects the instruction does not name put back removes the amplification on both tasks and on the robot.
| Comments: | 36 pages |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02527 [cs.RO] |
| (or arXiv:2610.02527v2 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02527 arXiv-issued DOI via DataCite |
Submission history
From: Jiaxuan Luo [view email]
[v1]
Thu, 1 Oct 2026 21:57:25 UTC (1,962 KB)
[v2]
Wed, 18 Nov 2026 00:11:01 UTC (1,152 KB)
来源:arXiv cs.AI · arxiv.org