arXiv cs.AI· Idriss Malek, Omar Choukrani, Daniil Orel, Anh Duy Le Dinh, Zhuohan Xie, Zangir Iklassov, Martin Tak\'a\v{c}, Salem Lahlou·· 3 天前AI 评分31
LLM-BabyBench：语言模型能否在可模拟的世界中做规划？
LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?
AI 导读
LLM-BabyBench 将程序生成的 BabyAI 网格世界改造为完全可观测的纯文本环境，把规划能力与感知、检索、常识等因素分离评估。其 PPD 套件包含 Predict、Plan、Decompose 三项任务，在七个前沿与开源模型零样本测试中，所有模型的模拟表现均远超规划，最强模型接近饱和。
正文 · 原文
Abstract:Interactive benchmarks for LLMs usually report a single metric, even though succeeding in them requires models to show several different skills. While those benchmarks are useful for measuring the overall capability of models to perform the downstream task, it is harder to identify which skill was lacking and caused failure solely from the aggregate score. For example, in benchmarks where models are supposed to navigate a simulated world to perform a task, failures can come from perception, an ambiguous instruction, retrieval, missing commonsense about what actions do, unknown dynamics, or planning. \benchmark recasts the procedurally generated BabyAI gridworld as a fully observable, purely textual environment designed to evaluate planning separately from those factors. The nature of the environment and the structure of the prompt control for the other factors cited above. To evaluate the models, we define the PPD suite: \textbf{Predict} asks for the state that follows an action sequence, \textbf{Plan} asks for an action sequence that reaches a goal, and \textbf{Decompose} asks for a subgoal sequence that achieves a mission. Across seven frontier and open models, queried zero-shot, we find simulation far ahead of planning for every model, close to saturated for the strongest. The length of the required solution governs planning failure, and each of the four strongest models has a horizon beyond which single-attempt success collapses. The responses of claude-opus-4-6 and claude-sonnet-4-6, which show their working step by step, reveal that they plan by proposing L-shaped corridors, which is consistent with their successful plans staying near-optimal while success falls. At the subgoal level, decomposition precision likewise falls to zero on long missions, while comprehension of those missions persists.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2505.12135 [cs.AI] |
| (or arXiv:2505.12135v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2505.12135 arXiv-issued DOI via DataCite |
Submission history
From: Idriss Malek [view email]
[v1]
Sat, 17 May 2025 20:23:17 UTC (240 KB)
[v2]
Fri, 11 Sep 2026 14:58:03 UTC (327 KB)
[v3]
Sun, 11 Oct 2026 16:59:10 UTC (326 KB)
来源：arXiv cs.AI · arxiv.org