arXiv cs.AI· Peisong Wen, Qianqian Xu, Minye Lei, Siran Dai, Yang Liu, Qingming Huang·· 12 小时前AI 评分31
Slot-JEPA：面向潜在世界模型的对象结构化预测接口
Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models
AI 导读
Slot-JEPA 将 JEPA 的稠密 patch 表示分解为稳定的语义 slot 与时变空间掩码，预训练一次并冻结后即可为下游提供可复用的对象结构化接口。配套的 Transition Memory Predictor（TMP）以紧凑掩码存储视觉转移，并用 Unbalanced Optimal Transport（UOT）监督掩码预测。
正文 · 原文
Abstract:As a prototypical latent world model, Joint-Embedding Predictive Architectures (JEPAs) provide strong semantic representations. However, dense patch representations expose a redundant interface to downstream predictors, which must implicitly infer spatial organization and temporal correspondence, presenting three challenges. First, object information is distributed across patches, forcing each downstream predictor to redundantly recover object structure that could otherwise be reused across tasks before modeling interactions. Second, the lack of explicit entity correspondence across time makes historical transitions expensive, restricting the predictor's flexibility in utilizing context. Third, without explicit transport structure, existing JEPA models typically use pointwise l1/l2 distances, which are insensitive to spatial displacement when predicted and goal supports are largely disjoint. We propose the Slot-based Joint-Embedding Predictive Architecture (Slot-JEPA), which decomposes dense representations into stable semantic slots and time-varying spatial masks. Pretrained once and frozen downstream, this decomposition exposes a reusable object-structured interface. Building on it, our Transition Memory Predictor (TMP) stores visual transitions as compact masks and retrieves useful context using statistics of corresponding slots. Its two branches separate slot-specific dynamics from shared cross-slot dynamics. We further use Unbalanced Optimal Transport (UOT) to supervise mask prediction and evaluate candidate plans by transporting mass between aligned predicted and goal masks, providing a signal better aligned with spatial progress. Across seven visual-understanding benchmarks, the slot-reconstructed features remain transferable. Slot-JEPA also improves goal-conditioned planning across six planar environments and four robotic arm planning tasks on the REALM-JEPA benchmark.
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.28074 [cs.AI] |
| (or arXiv:2610.28074v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28074 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Peisong Wen [view email]
[v1]
Tue, 20 Oct 2026 11:53:42 UTC (4,648 KB)
来源：arXiv cs.AI · arxiv.org