TwinRouterBench：面向真实智能体 LLM 路由的快速静态与实时动态评测基准
TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
TwinRouterBench 是一个步骤级 LLM 路由评测基准，分静态与动态两条赛道。静态赛道提供来自 SWE-bench、BFCL、mtRAG、QMSum 和 PinchBench 的 520 个实例共 970 条路由可见前缀，每条配有按公开降级级联协议估算并经执行验证的目标层级，评分仅基于层级标签、轨迹归属和 token 成本的确定性算术，无需在线 LLM 评审。
Authors:Pei Yang, Wanyi Chen, Tongyun Yang, Pengbin Feng, Jiarong Xing, Wentao Guo, Yuhang Yao, Yuhang Han, Hanchen Li, Xu Wang, Yuan Gao, Zeyu Wang, Jie Xiao, Anjie Yang, Liang Tian, Lynn Ai, Eric Yang, Tianyu Shi
Abstract:LLM routing matters most in long-horizon applications such as coding agents, deep research systems, and computer-use agents, where a single user request triggers many model calls. Routing each call to the cheapest sufficient model can cut costs without sacrificing quality, yet existing router benchmarks evaluate routers only on one-shot prompts. They never expose the router-visible prefix at an intermediate agent step, never test whether a cheaper replacement preserves downstream task success, and often rely on online LLM judges at evaluation time. We introduce TwinRouterBench, a step-level routing benchmark with two tracks. The static track provides 970 router-visible prefixes from 520 instances across SWE-bench, BFCL, mtRAG, QMSum, and PinchBench, each paired with an execution-verified target tier estimated under a released downgrade-and-cascade protocol; scoring is deterministic arithmetic over tier labels, trajectory membership, and token costs, with no online evaluator-side LLM judge. The dynamic track supplies a harness that runs routers on the full 500-case SWE-bench Verified suite; in this paper we report a 100-case held-out evaluation disjoint from the static SWE supervision split. At each LLM call the router selects a concrete model from a locked pool, and success is measured by official task resolution and realized API spend. The two tracks support fast offline iteration followed by end-to-end validation under live agent execution. Code and data are available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.18859 [cs.LG] |
| (or arXiv:2605.18859v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.18859 arXiv-issued DOI via DataCite |
Submission history
From: Pei Yang [view email]
[v1]
Thu, 14 May 2026 08:58:59 UTC (244 KB)
[v2]
Fri, 22 May 2026 02:55:33 UTC (348 KB)
[v3]
Tue, 29 Sep 2026 12:28:31 UTC (347 KB)
[v4]
Wed, 30 Sep 2026 04:49:28 UTC (347 KB)
[v5]
Sat, 24 Oct 2026 09:04:19 UTC (388 KB)
来源：arXiv cs.AI · arxiv.org