arXiv cs.AI· Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik, Or Litany, Yoni Kasten, Yuval Atzmon·· 13 小时前AI 评分42
Joints, Camera, Action!：为任意骨骼绑定联合生成 3D 运动与视频
Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs
AI 导读
研究者提出一种模型，可为任意骨骼绑定直接联合生成视频与 3D 运动，并给出将运动对齐到场景的相机。该模型将支持任意骨架的 3D 运动模型与预训练视频模型通过少量 cross-attention 层连接，先仅用运动数据学习骨架运动，再借关节在视频帧中的定位恢复相机。
正文 · 原文
Abstract:Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Graphics (cs.GR); Machine Learning (cs.LG); Robotics (cs.RO) |
| Cite as: | arXiv:2610.28632 [cs.CV] |
| (or arXiv:2610.28632v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28632 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Noam Rotstein [view email]
[v1]
Tue, 20 Oct 2026 17:59:56 UTC (6,339 KB)
来源：arXiv cs.AI · arxiv.org