arXiv cs.AI· Kailin Wang, Haoxiang Jie, Yaoyuan Yan, Zhiyou Heng, Zhaosong Li·· 4 小时前AI 评分24
EgoHandWAM:探究手部运动在以自我为中心的世界动作建模中的作用
EgoHandWAM: Probing the Role of Hand Motion in Egocentric World Action Modeling
AI 导读
EgoHandWAM 从当前图像和近期手部状态预测三维手腕与指尖轨迹及未来视觉特征,基于 751,065 个 EgoDex 窗口、用冻结的 DINOv2 编码器训练。
正文 · 原文
Abstract:How much does hand motion tell us about what a first-person camera will see next? EgoHandWAM forecasts three-dimensional wrist and fingertip trajectories and a future visual feature from the current image and recent hand states. We train on 751,065 EgoDex windows using a frozen DINOv2 encoder. On held-out development sessions, the action-only model reduces two-second wrist endpoint error from 60.26 to 56.48~mm relative to a stationary forecast. It still underestimates long reaches. Adding future-feature supervision gives nearly identical trajectory accuracy. With the recorded hand history, the visual head reduces smooth L1 error by 1.32\% relative to copying the current feature. Shuffling histories across sessions erases this gain. In a separate four-task probe, adding recorded future displacements to observed context and future tracking-validity masks reduces visual-feature error by 3.93\%. Masks alone give little improvement. A trajectory predicted from the same observations adds little. On two simulated RoboTwin tasks, substituting aligned human examples for an auxiliary batch of robot examples gives no consistent increase in task success across three training seeds.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22151 [cs.CV] |
| (or arXiv:2611.22151v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22151 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaoyuan Yan [view email]
[v1]
Tue, 17 Nov 2026 03:34:02 UTC (2,818 KB)
来源:arXiv cs.AI · arxiv.org