arXiv cs.AI· Yaru Niu, Binghong Chen, Changwei Yao, Shangtao Li, Zhenlong Fang, Revanth Krishna Senthilkumaran, Shuai Zhou, Yuemin Mao, Kalie Ching, Po-Ju Chao, Yudong Luo, Peiqi Yu, Heng Zhang, Hao Zhang, Bingqing Chen, Chen Qiu, H. Eric Tseng, Ye Yuan, Umar Iqbal, Changliu Liu, Jonathan Francis, Ding Zhao·· 1 天前AI 评分44
EgoX：用“具身 dreaming”实现第一人称跨具身操作
EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming
AI 导读
EgoX 是一个第一人称跨具身学习框架，通过语义对应的观测与动作模态组织异构演示，预训练模块化跨具身 Transformer MXT+ 并在目标具身上微调。在覆盖 5 个任务、6 种机器人具身（含 5 台实体机器人与 1 个仿真人形）的 13 个任务-具身组合上，跨具身预训练将平均归一化任务进度从 0.63 提升至 0.80，分布外条件下从 0.53 提升至 0.77。
正文 · 原文
Authors:Yaru Niu, Binghong Chen, Changwei Yao, Shangtao Li, Zhenlong Fang, Revanth Krishna Senthilkumaran, Shuai Zhou, Yuemin Mao, Kalie Ching, Po-Ju Chao, Yudong Luo, Peiqi Yu, Heng Zhang, Hao Zhang, Bingqing Chen, Chen Qiu, H. Eric Tseng, Ye Yuan, Umar Iqbal, Changliu Liu, Jonathan Francis, Ding Zhao
Abstract:Generalist manipulation policies can benefit from the complementary experience of humans, real robots, and simulated agents. However, differences in morphology, sensing, and control interfaces make it difficult to learn effectively from their combined demonstrations. We introduce EgoX, an egocentric cross-embodiment learning framework that organizes heterogeneous demonstrations through semantically corresponding observation and action modalities. EgoX pretrains a Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) on this mixture and fine-tunes it on target-embodiment demonstrations. Within each embodiment, MXT+ assigns a dedicated tokenizer to each observation modality and a dedicated expert to each action modality, with a Transformer trunk shared across embodiments. A training-only embodiment-dreaming objective predicts latent representations of future visual and proprioceptive states, encouraging the shared trunk to model each embodiment's dynamics. Across 13 task-embodiment pairs spanning five tasks and six robot embodiments, including five physical robots and one simulated humanoid, cross-embodiment pretraining raises MXT+'s average normalized task progress from 0.63 to 0.80 over training from scratch. The gains are particularly pronounced under out-of-distribution conditions, where average task progress increases from 0.53 to 0.77. Ablations demonstrate the benefits of broader pretraining mixtures, modular interfaces, and embodiment dreaming, with dreaming producing more similar latent geometry across embodiments. Project page: this http URL.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.16728 [cs.RO] |
| (or arXiv:2611.16728v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16728 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaru Niu [view email]
[v1]
Thu, 12 Nov 2026 12:40:24 UTC (24,155 KB)
来源：arXiv cs.AI · arxiv.org