AgentHop：面向科学引文图谱的智能体多跳问答诊断基准

View PDF HTML (experimental)

Abstract:Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources. Due to this complexity, agentic task failures arise from various sources, and pinpointing these failure causes is essential to diagnose and improve agentic systems. Existing benchmarks, however, tend to focus on a single leaderboard score, leaving the underlying failure modes opaque. To fill this gap, we introduce AgentHop, a diagnostic benchmark of 1,011 multiple-choice questions paired with a controlled seven-tool sandbox under fixed token, turn, and tool-call constraints. AgentHop reveals model vulnerabilities by dissecting a single accuracy score along four axes of agent operation: retrieval, synthesis, tool-call, and resource management. Across 19 models, we find that behavior clusters by model family, with tool-call signatures revealing distinct family fingerprints: GPT models commit early, Anthropic and GLM checkpoints verify before committing, DeepSeek and Kimi over-search, and Gemini-3 Pro stays balanced. Decomposed axes further expose within-family structure: Claude Opus 4.6 and Sonnet 4.6 land within one accuracy point yet diverge on retrieval-versus-synthesis emphasis, with Opus retrieving more and Sonnet synthesizing better. We release the full benchmark set and the harness to support diagnostic agent benchmarking.

Comments:	Accepted to NeurIPS 2026 Evaluations and Datasets Track
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2609.34428 [cs.CL]
	(or arXiv:2609.34428v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2609.34428 arXiv-issued DOI via DataCite

Submission history

From: Chanhee Park [view email]
[v1] Mon, 28 Sep 2026 06:45:01 UTC (1,016 KB)
[v2] Mon, 2 Nov 2026 03:48:35 UTC (1,017 KB)