arXiv cs.AI· Kwan Yan, Jialun Zhong, Yanzeng Li·· 1 天前AI 评分31
LEAP：具备可泛化长度预测的高效 LLM 推理调度系统
LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction
AI 导读
LEAP 是一个支持快速在线适配、在 CPU 上异步执行的长度预测系统，通过复用服务模型的 prefill 隐藏状态并用线性读出提取输出长度信号。在较大领域偏移下，其在线适配将 Spearman 相关系数提升 64.7%；服务实验中相比所评估的长度预测器降低 15.9–40.8% 的平均每 token 延迟，吞吐量高 1.9%，预测器参数量比最强基线少 30,569 倍。
正文 · 原文
Abstract:Length-aware scheduling reduces request latency in LLM serving by prioritizing shorter requests. However, despite their large model sizes, length predictors can suffer degraded ranking performance under domain and task shifts, and adapting them can require costly retraining. Their prediction overhead can further offset the benefits of scheduling. To improve the generalization and efficiency of length-aware scheduling, we propose \textbf{LEAP}, a length prediction system with fast online adaptation and asynchronous CPU execution. LEAP reuses the serving model's prefill hidden states, which contain information about the output length, and extracts this signal through a linear readout. To adapt to substantial domain and task shifts, it updates the readout at test time through recursive least squares, using completed requests as supervision without gradient computation or historical-data replay. Prediction and updates run asynchronously on CPU, while prediction and initial batch ordering overlap the remaining GPU prefill. Across the evaluated cross-domain transfers, LEAP retains strong ranking performance, and online adaptation improves Spearman correlation by 64.7\% under a larger domain shift. Serving experiments show that LEAP reduces average per-token latency by 15.9--40.8\% compared with the evaluated length predictors, while achieving 1.9\% higher throughput with $30{,}569\times$ fewer predictor parameters than the strongest evaluated baseline.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41359 [cs.AI] |
| (or arXiv:2610.41359v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41359 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kwan Yan [view email]
[v1]
Fri, 30 Oct 2026 07:23:27 UTC (300 KB)
来源：arXiv cs.AI · arxiv.org