arXiv cs.CL· Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana, Ammar Abbas, Solmaz Panahi, David Lynch, Zhuangzhuang Nie, Alexandros Agapitos, Aleksandar Milenovic, Hongmeng Song, Yucheng Shi, Yue Pan, Patricia Buffini, John D. Kelleher·· 12 小时前AI 评分34
CRE-Router：面向成本感知 LLM 服务的聚类、路由与升级级联框架
Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
AI 导读
CRE-Router（Cluster, Route, Escalate）是一种两阶段方法，先聚类查询并按 TPOT 或 E2EL 预算将每个簇路由到误差与成本权衡最优的模型，再由质量估计器在生成后决定是否升级到强模型。
正文 · 原文
Authors:Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana, Ammar Abbas, Solmaz Panahi, David Lynch, Zhuangzhuang Nie, Alexandros Agapitos, Aleksandar Milenovic, Hongmeng Song, Yucheng Shi, Yue Pan, Patricia Buffini, John D. Kelleher
Abstract:Serving open-weight language models on premises forces a trade-off between accuracy and computational cost. Hosting models of different sizes lets easy queries go to a small, efficient model and hard ones to a large model. We present CRE-Router (Cluster, Route, Escalate), a two-stage method that decides which of these models answers each query. It clusters queries and routes each cluster to the model that best trades error against cost, under a budget on the time per output token (TPOT) or the end-to-end latency per request (E2EL). It then either accepts the answer or, when a quality estimator rejects it, escalates the query to the strong model. Routing decides before generation, so a hard query goes straight to the strong model; escalation decides after, so it can catch the errors routing cannot predict. It needs no annotation beyond task-correctness labels. We evaluate CRE-Router on TeleMath, TeleQnA and AIME, with pools of two to nine models served on A100 GPUs. CRE-Router is more accurate than the efficient models in its pool and comparable to the strong model it escalates to, at a lower cost per token. We also find that the choice of cost term decides which models serve the workload, that pricing each model per cluster shortens the wait per request, and that a change of hardware changes the best routing.
| Comments: | Code: this https URL |
| Subjects: | Performance (cs.PF); Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.27457 [cs.PF] |
| (or arXiv:2606.27457v2 [cs.PF] for this version) | |
| https://doi.org/10.48550/arXiv.2606.27457 arXiv-issued DOI via DataCite |
Submission history
From: Yasmin Moslem [view email]
[v1]
Thu, 25 Jun 2026 18:29:24 UTC (141 KB)
[v2]
Fri, 23 Oct 2026 16:53:26 UTC (274 KB)
来源：arXiv cs.CL · arxiv.org