arXiv cs.CL· Shuyang Zhang·· 11 小时前AI 评分41
LLM 智能体多工具请求中检索结构的完整性与成本对比
Completeness and Cost of Retrieval Structures for Multi-Tool Requests in LLM Agents
AI 导读
研究对比六类工具检索结构在 AppWorld 345 条查询和 MCP-Bench 103 个任务（257 个 MCP 工具）上的表现，发现未在测试集训练且每查询最多一次 LLM 调用的结构在 k=16 时完成率均不足 10%。
正文 · 原文
Abstract:An agent that must call several tools to fulfill a request fails when the tool retriever omits any one of them, yet tool retrievers are mostly built and evaluated on queries requiring a single tool. We compare representatives of six retrieval structures (flat lexical and dense retrieval, LLM query decomposition, LLM-navigated hierarchies, server-then-tool matching, dependency graphs, and trained set-level retrievers) across two multi-tool testbeds: 345 AppWorld queries with expert dependency labels and 103 MCP-Bench tasks over 257 MCP tools. We measure completeness@k, the share of queries whose entire required tool set appears in the top k, alongside the LLM calls, tokens, and latency that each structure incurs at retrieval time. A motivating analysis on all 7,961 ToolRet queries shows that aggregate completeness mixes two regimes: completeness at 10 is .58 to .67 on single-target queries and .08 to .17 on queries with four or more targets, and the ordering of retrievers on single-target queries reverses on multi-target ones. Across the two testbeds, none of the compared structures that is untrained on the testbed and makes at most one LLM call per query completes more than 10% of AppWorld queries at k = 16. The two structures that do better incur different costs: a set-level retriever trained on 75 to 90 in-domain queries reaches .33 on AppWorld and .58 on MCP-Bench with no LLM calls, while an LLM-navigated hierarchy reaches .30 and .40 with a median of 31 to 38 LLM calls and about five minutes per query. Their failure modes also differ: the trained retriever mostly ranks the missing tools just below the cutoff, whereas the hierarchy finds the target API and stops before collecting its prerequisites. In a downstream check with the official AppWorld agent, tasks with a complete retrieved set succeeded 21 of 21 times, whereas incomplete ones succeeded 2 of 39 times.
| Comments: | 17 pages, 3 figures, 9 tables |
| Subjects: | Information Retrieval (cs.IR); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.41905 [cs.IR] |
| (or arXiv:2610.41905v1 [cs.IR] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41905 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuyang Zhang [view email]
[v1]
Fri, 30 Oct 2026 15:40:02 UTC (58 KB)
来源：arXiv cs.CL · arxiv.org