arXiv cs.AI· Minye Wu, Zehao Wang, Tinne Tuytelaars·· 9 天前AI 评分23
空间语言建模：统一策略学习与状态预测
Unifying Policy Learning and State Prediction through Spatial Language Modeling
AI 导读
研究者提出空间语言建模（Spatial Language Modeling），用离散坐标与语义 token 的共享词表表示场景轮廓、目标、动作目标和未来状态，让单个自回归 Transformer 通过统一的下一 token 预测目标同时学习动作生成与动作条件下的状态预测。
正文 · 原文
Abstract:Learning how actions change scene geometry can provide complementary supervision for goal-directed manipulation. We introduce Spatial Language Modeling, which represents scene contours, goals, action targets, and future states with a shared vocabulary of discrete coordinates and semantic tokens. A task-specific grammar organizes these elements into spatial sequences, allowing one autoregressive Transformer to learn action generation and action-conditioned state prediction through a common next-token objective. We train the model from scratch using random-play transition pretraining followed by joint action and state training on expert demonstrations. During pretraining, recorded action coordinates condition subsequent state predictions and are excluded from the prediction loss. During control, the model decodes only executable action targets and updates its history with newly observed states. We evaluate the approach on Push-T in simulation and on a real robot. The model achieves competitive simulation performance and higher task success and target coverage than the evaluated real-robot policy baselines. Training ablations show improved control with joint action and state sequences, with further gains from random-play pretraining. Given supplied action trajectories, the same model also predicts successive scene states, capturing the geometric effects of pushing.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.12172 [cs.RO] |
| (or arXiv:2610.12172v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12172 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Minye Wu [view email]
[v1]
Thu, 8 Oct 2026 15:43:08 UTC (4,403 KB)
来源：arXiv cs.AI · arxiv.org