arXiv cs.AI· Shawheen Naderi, Fatemeh Akherati, Yubei Chen·· 3 天前AI 评分32
PPO 智能体的价值函数中是否存在世界模型？可视化研究揭示 Crafter 环境中的概念学习
Visualizing the World Model in an Agent's Value Function
AI 导读
研究者对 PPO 智能体在 Crafter 环境中的价值函数施加反事实干预，发现尽管仅以优化奖励为目标训练，价值函数仍学到了钻石可收集规则、水与食物影响健康等概念。团队可视化这些世界模型并讨论了其安全启示，同时指出当前强化学习智能体阻碍构建真正先进机器智能的若干缺陷。
正文 · 原文
Abstract:Could model-free reinforcement learning algorithms actually be learning a model of the environment? In this work, we apply counterfactual interventions which hold everything about a state fixed, besides a targeted change, to explore if the value function of a Proximal Policy Optimization agent forms a world model of the Crafter environment. Despite only being trained to optimize reward, we discover that the value function learns concepts such as the rules governing when a diamond is collectible, water and food having an impact on health, heuristics regarding inventory, and more. Finally, we discuss the safety implications of our visualizations and some shortcomings of current reinforcement learning agents that will prevent us from building truly advanced machine intelligence.
| Comments: | 20 pages, 15 figues, 3 tables |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.14186 [cs.AI] |
| (or arXiv:2610.14186v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14186 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shawheen Naderi [view email]
[v1]
Fri, 9 Oct 2026 18:01:55 UTC (11,537 KB)
来源：arXiv cs.AI · arxiv.org