arXiv cs.AI· Lin Long, Tianren Ma, Zhong Chu, Di Wu, Maosong Cao, Qixiang Ye, Junbo Zhao, Shanghang Zhang·· 10 小时前AI 评分34
Ego Guidance：用无动作人类第一视角视频增强机器人策略学习
Ego Guidance: Enhancing Robot Policy Learning with Action-Free Human Egocentric Videos
AI 导读
研究者提出 SPAR（Shared Prediction on Action and Representation），一种基于预训练 VLM 的 MoT 架构，让未来视觉状态预测与动作预测并行提供独立互补的学习信号。实验显示，从第一视角视频学到的表征能提升机器人在分布外任务上的表现，且随视频数据规模扩大收益进一步提升。研究还考察了目标函数选择与数据筛选的影响。
正文 · 原文
Abstract:In this paper, we investigate how human egocentric videos can benefit robot policy learning without action supervision. Building on a pretrained VLM, we propose Shared Prediction on Action and Representation (SPAR), an MoT architecture in which future visual state prediction can provide independent and complementary learning signals alongside action prediction. We find that representations learned from egocentric videos improve downstream performance on out-of-distribution robot tasks, with further gains as video data scales. We also study the effects of objective choices and data curation, providing insights into how to leverage action-free videos for scalable robot learning.
| Comments: | Project Page: this https URL |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.13685 [cs.RO] |
| (or arXiv:2611.13685v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13685 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianren Ma [view email]
[v1]
Tue, 10 Nov 2026 07:33:35 UTC (2,512 KB)
来源：arXiv cs.AI · arxiv.org