arXiv cs.AI· Jeremy Collins, Namra Patel, Ayush Agarwal, Shitij Govil, Animesh Garg·· 11 小时前AI 评分42
FloMo：以 3D 场景流作为世界动作模型中间表示，从人类视频中学习
FloMo: 3D Scene Flow as a World Action Model Intermediate for Learning from Human Video
AI 导读
FloMo 是一个 5B 视频生成模型，经微调后预测 3D 场景流与动作 token，将场景流渲染为 RGB 并编码进与视频相同的 latent 空间，使预训练视频生成骨干能高效适配 3D 场景流预测。模型在无动作人类视频与带动作标注的机器人遥操作数据上联合训练，可泛化到未见物体、运动基元与任务。实验显示用视频预测目标替代运动预测会削弱分布外泛化，而同时加入视频预测目标反而降低策略成功率。
正文 · 原文
Abstract:Pretrained video generation models distill internet-scale motion priors into robot policies, but full video is a redundant signal for action prediction. We argue that dense 3D motion is the medium through which video pretraining best transfers to robot actions, and that predicting motion rather than video affords better sample-efficiency for action prediction. FloMo is a 5B video-generation model fine-tuned to predict 3D scene-flow and action tokens. Rendering scene flow as RGB and encoding it into the same latent space as video enables a pretrained video generation backbone to efficiently adapt to predict 3D scene flow. Co-trained on action-free human video and action-labeled robot teleoperation data, our model benefits from human-video pretraining and generalizes to unseen objects, motion primitives, and tasks. Replacing motion prediction with a video prediction objective degrades out-of-distribution generalization, while adding a video prediction target alongside motion lowers policy success rather than improving it. When motion, video, and action are predicted jointly, action tokens attend far more to motion than to video, suggesting that predicting video wastes model capacity. Together, these results indicate that motion effectively mediates the transfer from video pretraining to action. More information can be found at this http URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.05062 [cs.CV] |
| (or arXiv:2611.05062v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05062 arXiv-issued DOI via DataCite |
Submission history
From: Shitij Govil [view email]
[v1]
Tue, 3 Nov 2026 15:25:35 UTC (725 KB)
[v2]
Wed, 4 Nov 2026 03:57:24 UTC (725 KB)
来源：arXiv cs.AI · arxiv.org