arXiv cs.AI· Huajian Zeng, Lingyun Chen, Jiaqi Yang, Yuantai Zhang, Fan Shi, Peidong Liu, Xingxing Zuo·· 14 小时前AI 评分38
FlowHOI：基于流匹配的语义接地手物交互生成，用于灵巧机器人操作
FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation
AI 导读
FlowHOI 是一个两阶段流匹配框架，可根据第一视角观测、语言指令和 3DGS 场景重建生成语义接地的时序手物交互序列，包含手部姿态、物体姿态与接触状态。在 GRAB 和 HOT3D 基准上，其动作识别准确率最高，物理仿真成功率比最强扩散基线高 1.7 倍，推理速度提升 40 倍。研究还展示了在四项灵巧操作任务上的真机执行。
正文 · 原文
Abstract:Recent vision-language-action (VLA) models can generate plausible end-effector motions, yet they often fail in long-horizon, contact-rich tasks because the underlying hand-object interaction (HOI) structure is not explicitly represented. An embodiment-agnostic interaction representation that captures this structure would make manipulation behaviors easier to validate and transfer across robots. We propose FlowHOI, a two-stage flow-matching framework that generates semantically grounded, temporally coherent HOI sequences, comprising hand poses, object poses, and hand-object contact states, conditioned on an egocentric observation, a language instruction, and a 3D Gaussian splatting (3DGS) scene reconstruction. We decouple geometry-centric grasping from semantics-centric manipulation, conditioning the latter on compact 3D scene tokens and employing a motion-text alignment loss to semantically ground the generated interactions in both the physical scene layout and the language instruction. To address the scarcity of high-fidelity HOI supervision, we introduce a reconstruction pipeline that recovers aligned hand-object trajectories and meshes from large-scale egocentric videos, yielding an HOI prior for robust generation. Across the GRAB and HOT3D benchmarks, FlowHOI achieves the highest action recognition accuracy and a 1.7$\times$ higher physics simulation success rate than the strongest diffusion-based baseline, while delivering a 40$\times$ inference speedup. We further demonstrate real-robot execution on four dexterous manipulation tasks, illustrating the feasibility of retargeting generated HOI representations to real-robot execution pipelines.
| Comments: | Accepted by NeurIPS 2026. Project Page: this https URL |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2602.13444 [cs.RO] |
| (or arXiv:2602.13444v2 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2602.13444 arXiv-issued DOI via DataCite |
Submission history
From: Huajian Zeng [view email]
[v1]
Fri, 13 Feb 2026 20:46:08 UTC (19,164 KB)
[v2]
Mon, 26 Oct 2026 14:08:55 UTC (19,129 KB)
来源：arXiv cs.AI · arxiv.org