arXiv cs.AI· Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko, Mark C. Jeffrey·· 14 小时前AI 评分39
ArbiGraph：可任意扩展、可验证的任务图基准，用于评估上下文管理
ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management
AI 导读
ArbiGraph 是一个可生成可控任务图的基准生成器，用于评估工具辅助语言智能体能否在长推理流程中保留、更新、组合和丢弃任务相关上下文。其任务由自然语言问题与可执行 Python 求解器组成，长度、依赖结构、干扰项数量和值类型均可调，并保持精确自动验证。在数学、GSM 式应用题和 Python 追踪任务上评估 Qwen3.5-27B 工具智能体，依赖型分支链任务准确率最多下降 33.3%。
正文 · 原文
Abstract:We introduce ARBIGRAPH, a benchmark generator for evaluating whether tool-assisted language agents can retain, update, compose, and discard task-relevant context across extended reasoning workflows. ARBIGRAPH represents each task as a natural-language problem with an executable Python solver, and composes tasks through typed intermediate states, instantiated here as scalar and list values. This design enables controllable task graphs whose length, dependency structure, distractor count, and value type can be varied while preserving exact automatic verification. We instantiate ARBIGRAPH with math, GSM-style word-problems, and Python-tracing task categories, and evaluate a Qwen3.5-27B tool-assisted agent across four topologies. The results show high accuracy on isolated tasks but substantial degradation on more complex dependent tasks: accuracy drops by up to 33.3% on branching chains of dependent math tasks. This shows that ARBIGRAPH exposes failures that are not visible from single-task evaluation alone. Our code, generated datasets, and evaluation results are available at this https URL
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.20764 [cs.AI] |
| (or arXiv:2607.20764v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.20764 arXiv-issued DOI via DataCite |
Submission history
From: Pavel Golikov [view email]
[v1]
Wed, 22 Jul 2026 22:30:12 UTC (265 KB)
[v2]
Sat, 24 Oct 2026 08:19:20 UTC (193 KB)
来源：arXiv cs.AI · arxiv.org