arXiv cs.AI· Sukai Huang, Chenyuan Zhang, Fucai Ke, Zhixi Cai, Naim Rastgoo, Gholamreza Haffari, Hamid Rezatofighi·· 3 天前AI 评分31
LLM 规划能力研究：ACPBbench-Hard 上揭示两种不同的规划能力
What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities
AI 导读
研究在 ACPBench-Hard 上评估多个 LLM 系列，结合多维项目反应理论发现 LLM 规划表现由两种潜在能力主导：操作推理（评估局部动作适用性与即时状态转移）和结构枚举（推理目标可达性与地标结构）。操作推理随模型规模扩大和推理链变长而提升，结构枚举则相对不敏感。作者据此提出应转向能力层面的 LLM 规划评估。
正文 · 原文
Abstract:When LLMs exhibit uneven performance across planning tasks, these gaps are often attributed to task difficulty. We argue that this explanation is incomplete, as task-level variation may reflect distinct latent planning competencies rather than differences along a single ability spectrum. We study this question on ACPBench-Hard by evaluating multiple LLM families under varying test-time reasoning budgets and applying a multidimensional item response theory model to uncover the latent competency structure underlying LLM planning. The analysis reveals two principal dimensions that shape planning performance: operational reasoning, the ability to evaluate local action applicability and immediate state transitions, and structural enumeration, the ability to reason about goal reachability and landmark structure. Operational reasoning improving under model scaling and longer reasoning traces, while structural enumeration remains comparatively insensitive. Our findings motivate competency-level evaluation of LLM planning, shifting the focus from whether models improve overall to which planning competencies improve, under what conditions, and why.
| Comments: | Revised version after ARR rebuttal. Keywords: Reasoning, Automated Planning, Item Responses Theory, LLMs as Planner Research Area: NLP and Symbolic Reasoning Research Area Keywords: neurosymbolic, planning in agents, symbolic reasoning Contribution Types: Model analysis & interpretability |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.11197 [cs.AI] |
| (or arXiv:2607.11197v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.11197 arXiv-issued DOI via DataCite |
Submission history
From: Sukai Huang [view email]
[v1]
Mon, 13 Jul 2026 07:47:17 UTC (3,001 KB)
[v2]
Mon, 12 Oct 2026 18:16:00 UTC (2,918 KB)
来源：arXiv cs.AI · arxiv.org