arXiv cs.CL· Babak Barazandeh, Subhabrata Majumdar, George Michailidis·· 11 小时前AI 评分29
OTAP：用结构感知最优传输评估智能体轨迹中的规划与执行
OTAP: Structure-Aware Optimal Transport for Evaluating Planning and Execution in Agent Trajectories
AI 导读
研究者提出 OTAP（Optimal Transport for Agentic Planning），将智能体轨迹评估重构为执行图与有效解图集之间的距离，通过属性依赖图上的非平衡融合 Gromov-Wasserstein 传输问题实现，输出归一化差异分数。
正文 · 原文
Abstract:Large language model agents solve tasks by generating trajectories that interleave planning, tool calls, and intermediate results. Current evaluation metrics reduce such a trajectory to a binary success flag, compare it against a reference by exact matching, or delegate judgment to another language model. A success flag cannot distinguish sound solutions from lucky ones, nor explain why runs fail. Exact matching penalizes valid plans reordered or decomposed differently. We reframe trajectory evaluation as a distance between the agent's execution graph and a set of valid solution graphs, and instantiate it via an unbalanced fused Gromov-Wasserstein transport problem over attributed dependency graphs. The resulting score, termed OTAP (Optimal Transport for Agentic Planning), is a normalized discrepancy score. Its idealized, globally minimized entropy-free objective is invariant to dependency-preserving relabelings of a fixed graph and has a redundant-insertion sensitivity analysis under stated assumptions. Its unbalanced marginals handle missing or hallucinated steps without forcing a match; granularity invariance remains an open problem that soft coupling mitigates only in part. On controlled perturbations and three public benchmarks, OTAP separates valid from invalid trajectories in a regime where semantics-only metrics score below chance. Its performance is encouraging across these settings, while controlled edge-noise experiments reveal sensitivity to dependency extraction. Where a formal verifier exists, strict surface metrics predict validity better than OTAP does, which places OTAP in open-ended domains where no verifier is available. A blinded study of 236 trajectories finds the strongest agreement with OTAP among the six compared metrics. Structure-aware baselines achieve higher clean-graph AUROC, while OTAP has the smallest absolute AUROC decline under the tested edge noise.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.17082 [cs.AI] |
| (or arXiv:2607.17082v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.17082 arXiv-issued DOI via DataCite |
Submission history
From: Babak Barazandeh [view email]
[v1]
Sun, 19 Jul 2026 05:23:17 UTC (46 KB)
[v2]
Sat, 1 Aug 2026 14:51:11 UTC (37 KB)
[v3]
Tue, 3 Nov 2026 21:06:41 UTC (44 KB)
来源：arXiv cs.CL · arxiv.org