arXiv cs.AI· Shuang Luo, Yilun Kong, Yunpeng Qing, Yihang Jiao, Zhi Hou, Shunyu Liu, Xiaogang Wang, Dacheng Tao·· 9 天前AI 评分24
ACT³：一种以动作为中心的三流 Transformer，融合语义与动力学
Rewiring Semantics, Dynamics, and Control: A Simple yet Effective Action-Centric Tri-Stream Transformer
AI 导读
研究者提出 ACT³，一种以动作为中心的三流 Transformer，将 VLM 的语义理解与 WM 的动力学预测融合进控制动作，同时保持各上下文流的独立前向传播。该设计让专用动作专家通过逐层注意力访问 VLM 和 WM 表征，各骨干仅在本流内做注意力，并均可通过控制监督更新。在仿真与真实机器人操作基准上，ACT³ 表现优于同类方法。
正文 · 原文
Abstract:Vision-Language-Action (VLA) models have emerged as a prominent framework for complex robotic manipulation, building on the strong semantic understanding of pretrained Vision-Language Models (VLMs). However, such VLM backbones offer insufficient physical dynamics priors, which limits the generalization capabilities of robot policies. Recent efforts therefore integrate video-generation World Models (WMs) into robot policies through various strategies, using predictive dynamics to facilitate action generation. Despite these advances, harnessing semantic understanding and dynamics prediction as complementary guidance for action generation remains challenging. In this paper, we introduce $\mathrm{ACT}^3$, a simple yet effective Action-Centric Tri-Stream Transformer that fuses semantic and dynamics information into control actions while preserving the distinct roles of context streams. Specifically, $\mathrm{ACT}^3$ enables the dedicated action expert to access VLM and WM representations through layerwise attention, with each backbone attending only within its own stream. This straightforward interaction design maintains independent forward propagation in the context streams while allowing both backbones to be updated through control supervision. Experiments on both simulated and real-world robotic manipulation benchmarks show that the proposed $\mathrm{ACT}^3$ yields results superior to its counterparts.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11416 [cs.RO] |
| (or arXiv:2610.11416v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11416 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuang Luo [view email]
[v1]
Thu, 8 Oct 2026 07:45:58 UTC (2,742 KB)
来源：arXiv cs.AI · arxiv.org