arXiv cs.AI· Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou, Lingjun Mao, Ruobing Han, Jinglin Cao, Wenpeng Xu, Lukun He, Minghao Fu, Fan Feng, Biwei Huang·· 6 天前AI 评分37
Video2World：用具身视频评测编码智能体的交互式世界建模能力
Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
AI 导读
研究者提出 Video2World 基准，包含来自 189 段机器人与人类演示视频的 222 个重建实例，将"自主视频到仿真"定义为软件工程任务，从几何保真度、动态保真度和功能正确性三方面评测。在 9 个前沿编码智能体系统中，从 Claude Opus 5 开始任务成功率由不足 5% 升至超过 15%，但与人类辅助重建仍有明显差距。研究还发现视觉保真度更高并不总带来更高任务成功率。
正文 · 原文
Authors:Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou, Lingjun Mao, Ruobing Han, Jinglin Cao, Wenpeng Xu, Lukun He, Minghao Fu, Fan Feng, Biwei Huang
Abstract:Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate \emph{autonomous video-to-simulation} as a software engineering task in which an agent observes an embodied video, constructs the corresponding simulated environment and robot behavior, and iteratively refines the result through execution feedback. To evaluate this capability, we introduce \textbf{Video2World}, a benchmark comprising 222 reconstruction instances derived from 189 robot and human demonstration videos. Video2World measures reconstructed worlds along geometric fidelity, dynamic fidelity, and functional correctness, capturing spatial perception, physical reasoning, and executable interaction. Evaluating 9 frontier coding-agent systems reveals a sharp improvement in Task success beginning with Claude Opus 5, rising from below 5\% to over 15\%, while substantial gaps to human-assisted reconstruction remain. We further find that worlds that look better could work worse: better visual fidelity does not always lead to higher task success. This echoes the broader gap between perceptual realism and factual correctness observed in generative models.
| Comments: | Project page: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2610.04432 [cs.CV] |
| (or arXiv:2610.04432v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.04432 arXiv-issued DOI via DataCite |
Submission history
From: Jinzhou Tang [view email]
[v1]
Sat, 3 Oct 2026 10:46:52 UTC (42,988 KB)
[v2]
Wed, 7 Oct 2026 16:40:14 UTC (31,270 KB)
[v3]
Fri, 9 Oct 2026 05:21:02 UTC (31,270 KB)
来源：arXiv cs.AI · arxiv.org