arXiv cs.AI· Jiaao Yu, Yanjian Zhang, Haoyang Jiang, Guillaume Wisniewski, Nadi Tomeh, Shijie Jia·· 12 小时前AI 评分38
TouchéWorld：在生成式世界中实现统一的相机与手部在线探索
Touch\'{e}World tech report : Towards Online Unified Camera--Hand Exploration in Generative Worlds
AI 导读
TouchéWorld 是一个将手势驱动交互集成到相机控制视频世界模型中的框架，可根据初始图像和场景描述提示词，生成以时间对齐的相机与手部控制为条件的视频。该框架通过标注补全、一致性过滤和时间对齐，从异构第一人称视频构建联合相机-手部训练数据，并借助参数高效适配结合手部骨架空间条件与数值手部运动特征。研究团队还提出 TouchéBench，用于评估手部运动跟随、相机-手部协调、在线交互和物体响应。
正文 · 原文
Abstract:Recent video world models allow users to explore generated environments through camera navigation. Continuous hand motion provides a fine-grained action channel for interacting with the people and objects encountered during exploration. We present TouchéWorld, a framework that integrates gesture-driven interaction into a camera-controlled video world model. Given an initial image and a scene-description prompt, the model generates video conditioned on temporally aligned camera and hand controls. We construct joint camera--hand training data from heterogeneous egocentric videos through annotation completion, consistency-based filtering, and temporal alignment. Building on a pretrained camera-controlled world model, we combine spatial hand-skeleton conditioning with numerical hand-motion features through parameter-efficient adaptation. Causal adaptation and few-step distillation support chunk-level online generation, allowing users to update their controls based on generated visual feedback. We further introduce TouchéBench to evaluate hand-motion following, camera--hand coordination, online interaction, and object response. Together, these components establish a framework for studying continuous exploration and hand-driven interaction within generative worlds.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06061 [cs.CV] |
| (or arXiv:2611.06061v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06061 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiaao Yu [view email]
[v1]
Wed, 4 Nov 2026 07:06:54 UTC (26,472 KB)
来源：arXiv cs.AI · arxiv.org