arXiv cs.AI· Ma'ayan Armony, Albert Mero\~no-Pe\~nuela, Gerard Canal·· 13 小时前AI 评分34
LLM 距离符号规划器还有多远？一项基于 NLP 视角的评估
How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective
AI 导读
研究将 LLM 规划评估重构为 NLP 任务，用三步 NLP 流程加符号规划步骤细粒度衡量无效计划的质量。实验未发现 LLM 生成计划时存在逻辑推导或状态跟踪的证据：所有非空计划平均仅 3.55 个动作可执行，而符号生成的最优计划平均长 9.47 个动作，说明 LLM 仍远未达到符号规划器的可靠性。
正文 · 原文
Abstract:AI planning provides agents the ability to act autonomously towards achieving a goal in a given environment. Recent advances in Large Language Models (LLMs) have introduced the prospect of using unstructured input for planning, making LLMs in planning an active area of research, especially for supporting autonomous agents. However, LLMs remain unreliable as long-horizon planners, since some of the plans they produce contain mistaken or hallucinated actions; executing such plans could lead to unexpected and harmful outcomes. Existing benchmarks for LLMs in planning primarily rely on binary evaluation metrics, providing limited insight into the quality of plans that do not achieve the goal, and how far those are from becoming valid. In this paper, we frame the evaluation of LLMs in planning as a natural language processing (NLP) task, as LLMs are themselves NLP models. We propose a pipeline comprising three NLP-based steps that assess plan quality from complementary perspectives, followed by a symbolic planning step to achieve a valid plan, to provide a finer-grained understanding of the quality of invalid plans. In our experiments, we find no clear evidence of logical derivation or state-tracking during LLM plan generation -- on average, for all non-empty plans, only 3.55 actions per plan are executable, despite symbolically-generated optimal plans having an average length of 9.47 actions, which suggests that LLMs are still far from the reliability of symbolic planners. Our suite of metrics is further used to investigate patterns in LLM plan generation over time, and can provide a useful metric for assessing the quality of individual invalid plans.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2508.01300 [cs.AI] |
| (or arXiv:2508.01300v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2508.01300 arXiv-issued DOI via DataCite |
Submission history
From: Ma'ayan Armony [view email]
[v1]
Sat, 2 Aug 2025 10:20:52 UTC (243 KB)
[v2]
Tue, 20 Oct 2026 09:27:59 UTC (126 KB)
来源：arXiv cs.AI · arxiv.org