arXiv cs.AI· Boxi Zhou, German Sviridov, Dmitrii Ustiugov, Haris Javaid·· 10 小时前AI 评分42
用超时级联降低编程智能体推理成本
Reducing Inference Cost of Coding Agents with Timeout Cascades
AI 导读
研究评估了 2024 年以来发布的 36 个开源大、小模型,发现近期小模型在软件工程基准上正缩小与大模型的准确率差距,且差距高度集中在难题上。采用先试小模型、必要时升级到大模型的混合推理策略,总 GPU 小时成本最多降低 33%,在 SWE-bench Verified 全部任务上成本降低 16.4% 且不牺牲准确率。该简单级联方案接近 oracle 式任务路由器的效果,无需复杂的学习型路由器。
正文 · 原文
Abstract:AI coding agents have become one of the most popular applications of large language models. Yet, agentic AI comes at the cost of a tremendous amount of GPU compute resources, which limits its profitability and its viability at scale. In this paper, we investigate whether smaller, cheaper models can cut this cost without sacrificing accuracy.
We evaluate 36 popular large and small open-weight models released since 2024 on a software engineering benchmark, and find that recent small models are closing the accuracy gap compared to their large counterparts. We investigate the structure of this gap and find that the gap is highly concentrated around hard tasks that small language models fail to solve while achieving remarkably good accuracy for easy and medium complexity tasks. Following this observation, we show that by utilizing a hybrid inference strategy that tries a small model first and escalates to a large one only when needed, we can achieve up to 33% reduction in total GPU-hour costs and 16.4% on all tasks from SWE-bench Verified, without sacrificing accuracy. Strikingly, this simple cascade nearly matches an oracle-style task router, highlighting a practical path for deploying cost-aware coding agents in production without complex learned task routers and with minimal engineering complexity.
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.24324 [cs.AI] |
| (or arXiv:2611.24324v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24324 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: German Sviridov [view email]
[v1]
Wed, 18 Nov 2026 12:26:42 UTC (180 KB)
来源:arXiv cs.AI · arxiv.org