arXiv cs.AI· Yifeng Yao, Jinghan Li, Zhihao Zhang, Yajiao Xiong, Yang Jin, Kun Xu, Huishuai Zhang, Dongyan Zhao·· 12 小时前AI 评分38
VANE：在嵌入空间做下一帧预测的统一视觉世界模型预训练框架
Learning from Next-Frame Prediction: Towards Unified and Scalable Visual World Modeling
AI 导读
研究者提出自监督视觉预训练框架 VANE，在冻结语义 tokenizer 提供的固定嵌入空间中用因果编码器总结过去观测、由状态预测器推断整帧表示，并以轻量生成解码器重建未来帧 patch 保留细节。
正文 · 原文
Abstract:Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pretraining focuses on recovering missing content from surrounding context rather than predicting the future from the past. Autoregressive methods instead predict forward one token at a time, but have yet to match joint-embedding methods in representation quality. This gap motivates us to revisit the granularity and space of future prediction. In this work, we introduce VANE, a self-supervised visual pretraining framework built on next-frame prediction in embedding space. A causal encoder summarizes past observations for a state predictor that infers the representation of the entire next frame, making temporal evolution the primary learning signal. This prediction is carried out in a fixed embedding space provided by a frozen semantic tokenizer, emphasizing scene content and structure. To retain fine-grained spatial detail alongside semantic content, a lightweight auxiliary generative decoder reconstructs future-frame patches conditioned on the predicted representations. To unify visual modeling across images and videos, we construct a local-to-global crop sequence for still images, allowing the same encoder to predict newly revealed regions from earlier crops under the same objective. Experiments show that VANE improves with model and data scaling, achieving SOTA on most anticipation, recognition, and dense perception benchmarks among unified encoders. Especially on EPIC-KITCHENS-100 action anticipation, it surpasses V-JEPA 2.1 by 7.1 points in mean-class recall@5. These results demonstrate that next-frame prediction in embedding space is effective and scalable for visual world modeling.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.30724 [cs.CV] |
| (or arXiv:2610.30724v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30724 arXiv-issued DOI via DataCite |
Submission history
From: Yifeng Yao [view email]
[v1]
Thu, 22 Oct 2026 04:46:49 UTC (3,474 KB)
[v2]
Fri, 23 Oct 2026 03:33:22 UTC (3,474 KB)
来源：arXiv cs.AI · arxiv.org