arXiv cs.AI· Randall Balestriero, Yann LeCun, Kerda Varaku·· 10 小时前AI 评分47
反事实世界模型:能推理“如果……会怎样”的 JEPA
Counterfactual World Models: JEPAs That Reason About "What If"
AI 导读
研究证明,仅用事实性转移数据训练的世界模型无法自行回答反事实问题,但利用“自然不跳跃”性质可推导出反事实结果,无需反事实数据、额外参数或重训练。基于冻结 DINOv2 与 MAE 编码器的 JEPA 世界模型在四个渲染模拟器上将反事实 R² 从 38%–46% 提升至 86%–96%,第五个水下航行器场景达 90%–94%。
正文 · 原文
Abstract:World models operate in a world they cannot fully observe. When a hidden disturbance, such as a current pushing a boat, changes the outcome of an action, a counterfactual question follows: what would have happened under a different action, subject to the same disturbance? Its answer separates bad decisions from bad luck. We prove that world models trained on factual transitions cannot answer it on their own, whatever their data or capacity. For many physical systems, the observed outcome and a simple property of the dynamics suffice: natura non facit saltus, nature makes no leaps. When actions and disturbances act as separate forces, their interaction is second-order in time, so the counterfactual outcome is, to first order, the observed outcome plus the difference between the world model's predictions under the alternative and factual actions. This yields a counterfactual world model with no counterfactual data, extra parameters, or retraining. For transitions of duration $h$, we prove an $O(h^2)$ bound on the counterfactual error relative to the disturbance variance, establish a matching minimax lower bound from unpaired factual data, and characterize failures at discontinuities, coordinate folds, and disturbances that scale the command. We validate these predictions from pixels with JEPA world models on frozen DINOv2 and masked autoencoder (MAE) encoders. On four rendered simulators, our approach improves counterfactual $R^2$ from $38$--$46\%$ to $86$--$96\%$ with the same world models; on a fifth, an underwater vehicle, it reaches $90$--$94\%$ with sixteen times more factual data. Four further frozen encoders, DINOv3, SigLIP~2, CLIP and EVA-02, give $81$--$95\%$ on the same simulators. In hindsight planning, it finds the action that would have reached the goal in $35$--$69\%$ of events, against $14$--$23\%$ for an oracle planner that knows the simulator but not the outcome.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO); Machine Learning (stat.ML) |
| Cite as: | arXiv:2611.24778 [cs.LG] |
| (or arXiv:2611.24778v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24778 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Randall Balestriero [view email]
[v1]
Wed, 18 Nov 2026 17:24:00 UTC (1,374 KB)
来源:arXiv cs.AI · arxiv.org