arXiv cs.AI· Guoqing Wang, Pin Tang, Xiangxuan Ren, Chao Ma·· 23 小时前AI 评分34
AgentWorld:面向自动驾驶的智能体中心 VLA 世界模型
Learning Agent-Centric Vision-Language-Action World Models for Autonomous Driving
AI 导读
研究者提出 AgentWorld,一种以智能体为中心的 VLA 世界模型,用离散 agent token 同时进行推理与轨迹规划,每个 token 动态绑定到周围智能体的语义身份、3D 几何、运动与交互上下文。
正文 · 原文
Abstract:Recent progress in integrating vision-language-action (VLA) models with world modeling has significantly advanced end-to-end autonomous driving by allowing systems to reason about imagined futures instead of merely reacting to current observations. However, most world VLA models generate future frames at the pixel level with plenty of dense visual tokens. Although such generated futures can be visually plausible, they preserve many driving-irrelevant appearance details, i.e., only a small portion of the tokens directly encode critical planning-relevant factors, such as agent geometry, motion dynamics, and interactions. This introduces a significant performance bottleneck. An important question thus arises: what world state should a world VLA model reason over to support planning beyond pixel-level prediction? We argue that planning-oriented world modeling should focus on a compact and structured agent-centric latent state that captures the interactions between objects in the driving scene over time. As such, we introduce AgentWorld, an agent-centric VLA world model that uses discrete agent tokens jointly for reasoning and trajectory planning. Each token is dynamically grounded to a surrounding agent associated with its object's state, such as semantic identity, 3D geometry, motion, and interaction context. AgentWorld learns these tokens through three key processes: explicit geometric grounding, reasoning-oriented supervised fine-tuning, and agent-oriented reinforcement learning. Extensive open-loop and closed-loop experiments under diverse driving scenarios show that AgentWorld enhances planning capabilities, fully exploits visual evidence and agent dynamics, and yields interpretable agent-centric world states.
| Comments: | Accepted by NeurIPS 2026 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22880 [cs.CV] |
| (or arXiv:2611.22880v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22880 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guoqing Wang [view email]
[v1]
Tue, 17 Nov 2026 14:53:49 UTC (5,456 KB)
来源:arXiv cs.AI · arxiv.org