arXiv cs.AI· Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn, Krishnan Srinivasan, Animesh Garg, Vitor Campagnolo Guizilini, Paarth Shah·· 2 天前AI 评分38
TAPESTRY：用单一骨干网络同时做视频生成与机器人控制
TAPESTRY: A Single Backbone for Video Generation and Robot Control
AI 导读
TAPESTRY 将预训练视频扩散 Transformer 复用为机器人策略，把每个动作时间步映射为视频 VAE 潜空间中的单 patch token，与视频共用同一套 patchify 层和 Transformer 骨干，推理时无需生成视频。
正文 · 原文
Abstract:Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at this https URL.
| Comments: | Accepted to Conference on Robot Learning (CoRL) 2026 |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.36823 [cs.RO] |
| (or arXiv:2610.36823v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36823 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jeremy Collins [view email]
[v1]
Tue, 27 Oct 2026 08:23:30 UTC (3,146 KB)
来源：arXiv cs.AI · arxiv.org