arXiv cs.AI· Selin Yildirim, Ali Zeyrek, Deming Chen·· 4 小时前AI 评分31
SpecSearch:面向硬件感知的高效投机解码
SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding
AI 导读
SpecSearch 是一个硬件感知搜索引警,联合优化解码策略、草稿长度与精度,在 RTX 5090、H100、L40 和 AMD MI300X 上配合 vLLM 实现平均 2.01 倍于自回归解码、1.36 倍于投机解码的吞吐量。
正文 · 原文
Abstract:Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX~5090, the state-of-the-art speculative drafter DFlash achieves $3.50\times$ the throughput of autoregressive (AR) decoding at low concurrency, but only $0.76\times$ at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload--GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX~5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of $2.01\times$ the throughput of AR and $1.36\times$ the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with $1.30\times$ throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains $97.36\%$ of the throughput achieved by SpecSearch that utilizes full calibration.
| Subjects: | Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2611.22296 [cs.AI] |
| (or arXiv:2611.22296v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22296 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Selin Yildirim [view email]
[v1]
Tue, 17 Nov 2026 06:53:30 UTC (110 KB)
来源:arXiv cs.AI · arxiv.org