arXiv cs.AI· Shivam Patel, Akaash R. Parthasarathy, Akash Dhasade, Ankur Mallick, Gauri Joshi·· 2 天前AI 评分22
Mind the Delay：面向异构 LLM 池的延迟感知查询路由
Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools
AI 导读
研究者提出 Batch Evolution Aware（BEA）估计器，通过重放服务框架的 token-batch 形成过程预测首 token 延迟（TTFT），误差约 5%，单次预测耗时约 0.1 ms。基于 BEA 的延迟感知路由器 BEAR 同时平衡准确率、成本与延迟，在七项基线对比中，目标内响应效用平均提升 63%，较最佳基线提升 30.5%。
正文 · 原文
Abstract:Large language model (LLM) query routers typically assign each query to a suitable model that balances response accuracy and monetary cost. Subsequently, separate system-level load balancers based on policies such as join-the-shortest-queue or round-robin distribute the queries across replicas of each model to reduce latency. However, decoupling accuracy-cost and latency considerations can lead to suboptimal routing decisions. We introduce the Batch Evolution Aware (BEA) estimator to provide a fine-grained estimate of the latency, that is, the time-to-first-token (TTFT) experienced by an arriving query. Unlike existing load balancers that rely on coarse signals such as queue lengths, per-model processing rates, or historical average latencies, BEA replays the serving framework's token-batch formation using a snapshot of an instance's current workload, stopping when the incoming query's first token is generated. Taking into account the compute and memory requirements of the prefill and decode phases of language queries as well as the contention between multiple queries that are served in the same batch, BEA predicts TTFT within an error of approximately 5%. Each prediction takes about 0.1 ms, which is fast enough to run on the critical routing path. We also design an analytical framework for a model instance under a limited processor-sharing scheme that calculates expected TTFT in closed form and captures the qualitative trend of latencies as the load increases. The TTFT estimates produced by BEA are then used in our latency-aware router, BEAR, which simultaneously balances accuracy, cost, and latency to select the best model instance from a heterogeneous pool of LLMs. Against seven baselines, BEAR improves the utility of responses delivered within their targets by 63% on average over existing latency-aware routers, and by 30.5% over the best baseline at each arrival rate.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.18253 [cs.AI] |
| (or arXiv:2607.18253v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.18253 arXiv-issued DOI via DataCite |
Submission history
From: Shivam Patel [view email]
[v1]
Wed, 13 May 2026 20:29:09 UTC (1,247 KB)
[v2]
Thu, 15 Oct 2026 02:47:09 UTC (1,548 KB)
来源：arXiv cs.AI · arxiv.org