跳到正文
arXiv cs.AI· Tian Lyu, Yichen Jiang, Ziyuan Cheng, Yunda Yan, Xiangyu Pan, Junjie Jiang, Jialin Yu, Boyu Ning, Pietro Li\`o, Fan Mo·· 4 小时前AI 评分35

JointGV:用并行图动态为视觉世界模型做结构感知校正,提升长程建模能力

JointGV: Structure-Aware Correction of Visual World Models with Parallel Graph Dynamics for Long-Horizon Modeling

AI 导读

JointGV 是一种可插拔的图-视觉动态校正模块,通过并行推演任务条件场景图并耦合图节点与视觉状态的变化,为视觉世界模型计算有界残差以纠正长程漂移。在 DINO-WM 与 DreamerV3 上,32 步推演将关节位置误差降低 11–28%;真实机器人倒立摆序列中,DINO-WM 的 32 步杆角误差从 12.56° 降至 6.69°。

正文 · 原文

View PDF HTML (experimental)

Abstract:Visual world models predict future visual states conditioned on actions, but recursive errors in entity poses and spatial relations limit their long-horizon prediction for robot planning and control. We ask whether a separately predicted relational state can correct this drift without restructuring the visual model. This is challenging because graph nodes and visual states share neither elements nor a feature space. We propose Joint Graph--Visual Dynamics Correction (JointGV), a plug-in based on a cross-granularity dynamics hypothesis: despite their different representations, graph-node and visual-state changes describe the same physical transition. JointGV rolls out a task-conditioned scene graph alongside the visual world model, couples their predicted changes, and combines the resulting graph-motion context with graph-predicted node positions to compute a bounded residual. Graph-learning and correction gradients do not propagate into the original visual dynamics. We attach JointGV to DINO-WM (spatial tokens) and DreamerV3 (a global latent state), evaluating both on DMC Cheetah Run and Walker Walk across three seeds and extending the evaluation to contact-rich manipulation with DMC Finger Turn Hard. Across 32-step rollouts, JointGV improves structural and visual-prediction accuracy, reducing joint-position error by 11--28\%. For spatial tokens, learned graph--visual change coupling gives lower error than broadcasting the same mean graph-motion context to every token, particularly when non-adjacent parts appear close together in the image. On real-robot inverted-pendulum sequences, JointGV reduces DINO-WM's 32-step pole-angle error from $12.56^\circ$ to $6.69^\circ$ and improves DreamerV3 and LeWorldModel as well. In simulated inverted-pendulum control, an actor trained with JointGV-corrected rollouts completes 46 of 50 episodes, compared with none using the original rollouts.
Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2611.22543 [cs.RO]
  (or arXiv:2611.22543v1 [cs.RO] for this version)
  https://doi.org/10.48550/arXiv.2611.22543

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Tian Lyu [view email]
[v1] Tue, 17 Nov 2026 10:33:21 UTC (5,777 KB)

来源:arXiv cs.AI · arxiv.org