arXiv cs.AI· Yaxin Ma, Wenbo Zhang, Satya Krishna Pothapragada, Alina Zare·· 2 天前AI 评分27
GraphACM：基于图的动作一致性度量，用于 LLM 智能体的不确定性量化
GraphACM: Graph-based Action Consistency Measurement for Uncertainty Quantification in LLM Agents
AI 导读
研究者提出 GraphACM，一种基于图的框架，通过将同一任务多次 rollout 中的等价状态合并为有向动作标注图，在共享决策状态上比较动作，实现细粒度行为一致性度量与不确定性量化。
正文 · 原文
Abstract:Behavioral consistency across repeated executions provides an informative signal for assessing the reliability of large language model (LLM) agents. However, existing approaches typically summarize consistency with a single task-level score, without identifying where behavioral differences occur or whether actions are compared under equivalent decision contexts. We propose a graph-based framework for fine-grained behavioral consistency measurement and uncertainty quantification. Given multiple rollouts of the same task, we merge equivalent states into a directed, action-labeled graph, enabling action comparisons at shared decision states. We introduce node-level and global measures of action consistency, using the global consistency score as a proxy for uncertainty to predict task success or failure. Experiments across three LLMs on ALFWorld, AppWorld, and WebShop demonstrate that our measure is positively associated with task success and achieves an average AUROC of 0.811 for task-success prediction, outperforming the strongest behavioral-consistency baseline (0.729) and verbalized confidence (0.769). These results highlight the effectiveness of state-conditioned behavioral consistency and pinpoint where agent actions agree.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.22031 [cs.AI] |
| (or arXiv:2610.22031v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22031 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaxin Ma [view email]
[v1]
Thu, 15 Oct 2026 15:07:15 UTC (534 KB)
来源：arXiv cs.AI · arxiv.org