arXiv cs.AI· Nicholas Saban, Charvi Bannur, Mariya Vasileva, Darshan Deshpande·· 12 小时前AI 评分38
GlobeBench：评估文本世界模型的可扩展环境模拟能力
GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations
AI 导读
GlobeBench 是一个包含 1,632 个任务、覆盖五个领域和十九项模拟技能的文本世界模型基准，用于检验模型能否根据完整智能体轨迹预测环境对特定动作的响应。在 20 个通用 LLM 和专用文本世界模型中，GPT-6 Astra 任务成功率最高，为 49.57%；当动作影响需传播到依赖对象时，20 个模型平均成功率降至 5.1%，另有 631 个任务无任何模型解决。
正文 · 原文
Abstract:Text-based world models have recently shown potential to simulate complex RL environment state transitions for a diverse set of use cases. However, no standardized benchmark exists to compare, study and improve these models. Through this paper, we attempt to solve this problem by building GlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills. The tasks check whether models can predict the environment's response to a specific action given the full agent trajectory by understanding earlier state changes, following environment rules, and predicting the effects of actions on both related and unrelated objects. We compare the predicted environment response with an execution-derived reference, using a hybrid architecture with both deterministic gates and a semantic judge. Among 20 general-purpose LLMs and specialized text world models, GPT-6 Astra achieves the highest task success rate at 49.57%. We find that model accuracy decreases as actions affect a larger portion of the environment. Specifically, when the agent's changes must propagate to dependent objects, task success drops to an average of 5.1% across all 20 models. Although model strengths differ across tasks, 631 tasks remain unsolved by any model, suggesting a significant gap in general LLM capabilities as a world model.
| Comments: | Code: this https URL ; Data: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22708 [cs.AI] |
| (or arXiv:2610.22708v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22708 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nicholas Saban [view email]
[v1]
Thu, 15 Oct 2026 22:24:56 UTC (300 KB)
来源：arXiv cs.AI · arxiv.org