arXiv cs.AI· Jiayi Luo, Yang Zhang, Liliang Chen, Nan Wang, Chen Gao, Boxiang Qiu, Zhuoxuan Li, Baorui Peng, Chuankang Li, Yuxiang Yan, Si Liu, Qingyun Sun, Shuicheng Yan, Maoqing Yao, Jianxin Li, Guanghui Ren·· 13 小时前AI 评分48
Rollback：在世界模型中回滚经验以改进策略
Rollback: Experience Backtracking in World Models for Policy Improvement
AI 导读
Rollback 提出一种策略改进范式，把世界模型从单向模拟器变成反事实引擎：回滚到关键决策点、分支尝试不同动作并对比结果，将反事实结果蒸馏为动作级监督，无需单独学习价值模型。在真实机器人任务上，其成功率较最强基线分别绝对提升 10.67%、11.11% 和 11.85%（方块抽取、抛球、桌面冰壶），并在抛球和桌面冰壶上训练 8 小时内超过人类玩家平均成功率。
正文 · 原文
Authors:Jiayi Luo, Yang Zhang, Liliang Chen, Nan Wang, Chen Gao, Boxiang Qiu, Zhuoxuan Li, Baorui Peng, Chuankang Li, Yuxiang Yan, Si Liu, Qingyun Sun, Shuicheng Yan, Maoqing Yao, Jianxin Li, Guanghui Ren
Abstract:Action-conditioned world models offer a scalable alternative to costly real-world interaction for improving robot policies. However, existing world-model-based methods largely treat world models as forward-only surrogates of the physical environment, leaving untapped a capability no physical environment affords: rewinding to past decision points to explore counterfactual futures. We introduce Rollback, a policy improvement paradigm that turns world models from passive simulators into counterfactual engines: it rolls back to critical decision points, branches into alternative actions, and contrasts their outcomes to attribute success to individual decisions. This concentrates imagination where it matters most and distills counterfactual outcomes into fine-grained, action-level supervision without a separately learned value model. We build a scalable training system that unifies distributed world-model search with asynchronous real-robot execution, forming a self-reinforcing cycle: real-world experience continually sharpens the world model, and a sharper world model in turn drives further policy improvement. We term the world-model side of this cycle test-time physics grounding, a capability that hand-built physics simulators lack: while the world model serves as the training environment, real transitions from the policy it trains correct its dynamics through a lightweight LoRA pipeline. Experiments on three challenging real-world tasks show that Rollback improves success rates over the strongest baseline by an absolute 10.67%, 11.11%, and 11.85% on block extraction, ball tossing, and tabletop curling, respectively, while achieving more stable improvement throughout training. Although initialized from teleoperated demonstrations far below human level, Rollback surpasses the average success rate of human players on both ball tossing and tabletop curling within 8 hours of training.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.36068 [cs.RO] |
| (or arXiv:2610.36068v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36068 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiayi Luo [view email]
[v1]
Mon, 26 Oct 2026 18:01:02 UTC (15,362 KB)
来源：arXiv cs.AI · arxiv.org