Thinkingbox 发布：面向有状态业务流程智能体的沙盒与基准
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
论文提出 Thinkingbox，一个支持工具、智能体与用户交互的沙盒，提供隔离的 MCP 兼容工具会话、完整执行轨迹和基于后端终态的评估；在其上构建的 Thinkingbox-bench 包含零售、酒店、车险、数字银行内部 IT、咨询 IT/HR 支持等场景的 507 个策略约束工作流。
Authors:Zhuochun Li, Youngmin Ko, Ali Keramati, Tuhin Kundu, Liang-Chun Tsai, Nicola Ferri, Mirco Milletari, Jiaxiang Liu, Susana Palmaz Lopez Pelaez, Yuepeng Wang, Vadim Smolyakov, Xiang Jiang, Kjartan Olafsson, Tommy Guy
Abstract:Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across business scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Our experiments reveal that even the strongest proprietary and open-weight models show steep reliability drops: Claude Opus 5 falls from 66.50% pass@1 to 47.53% pass^20, and Kimi-K3 from 57.37% pass@1 to 17.60% pass^20. Moreover, many failed trials terminate cleanly after valid state-changing actions, so response- or tool-call-level signals poorly proxy end-to-end completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release Thinkingbox, Thinkingbox-Bench, and Thinkingbox-integrated reinforcement learning framework.
| Subjects: | Computation and Language (cs.CL); Databases (cs.DB) |
| Cite as: | arXiv:2608.19741 [cs.CL] |
| (or arXiv:2608.19741v5 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.19741 arXiv-issued DOI via DataCite |
Submission history
From: Zhuochun Li [view email]
[v1]
Thu, 20 Aug 2026 07:37:57 UTC (2,548 KB)
[v2]
Sat, 29 Aug 2026 00:45:56 UTC (5,511 KB)
[v3]
Wed, 30 Sep 2026 00:33:55 UTC (3,355 KB)
[v4]
Thu, 1 Oct 2026 16:14:54 UTC (3,355 KB)
[v5]
Sat, 31 Oct 2026 00:08:26 UTC (3,354 KB)
来源：arXiv cs.CL · arxiv.org