arXiv cs.AI· Jiachun Jin, Zetong Zhou, Haonan Wang, Xiao Yang, Jun Zhu, Zhijie Deng·· 1 天前AI 评分38
SALA：语义对齐的潜在动作实现具身基础模型双轴扩展
SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models
AI 导读
SALA 通过无配对数据的方式，利用人类与机器人视频片段的语言描述估计语义相似度，监督跨具身对应关系，解决现有潜在动作模型（LAMs）按视觉外观而非动作聚类的问题。该方法构建共享潜在动作空间，在预训练阶段用更多通用人类视频持续降低下游机器人动作预测误差，在后训练阶段用更多任务特定人类视频显著提升真实机器人任务成功率。
正文 · 原文
Abstract:Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However, most existing LAMs learn latent actions solely through video reconstruction, which can cause their representations to cluster by visual appearance rather than by the underlying actions. As a result, they struggle to align human actions with their robot counterparts. To bridge the gap, we introduce SALA, which emphasizes establishing the semantic correspondence for latent actions. Without requiring paired data, we use the language descriptions of unpaired human and robot clips to estimate semantic similarities and supervise cross-embodiment correspondence. Specifically, we translate these similarities into soft neighborhood structures that guide human latent actions toward semantically related robot latents, while robot action reconstruction grounds the latent space in control. The resulting shared latent action space provides a unified supervision signal for training an embodied foundation model on both human and robot videos. Notably, latent action supervision enables scaling along two axes: (1) pre-training on increasing amounts of general human video consistently reduces robot action prediction error on downstream tasks, and (2) post-training on increasing amounts of task-specific human video substantially improves real-world robot task success rates.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.17110 [cs.RO] |
| (or arXiv:2611.17110v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17110 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiachun Jin [view email]
[v1]
Thu, 12 Nov 2026 17:04:50 UTC (12,741 KB)
来源：arXiv cs.AI · arxiv.org