跳到正文
arXiv cs.AI· Zewen Huang, Xiangwen Zhuge, Yingkun Hu, Zheng Yang, Qiang Ma·· 4 小时前AI 评分32

HyCoInfer:面向异构 GPU-FPGA 架构的云边协同 LLM 推理系统

HyCoInfer: Cloud-Edge Collaborative Inference for Large Language Models on Heterogeneous GPU-FPGA Architectures

AI 导读

HyCoInfer 是一个将 prefill 放在云端 GPU、把解码分散到云端 GPU 与边缘 FPGA 的异构云边推理系统,在 500 Mbps 链路上实现最高达纯云端 vLLM 4.11 倍的请求吞吐量,平均端到端延迟最多降低 61.1%。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large language model (LLM) serving relies predominantly on cloud GPUs, yet autoregressive decoding is memory-bound and often leaves their arithmetic units underutilized. Edge FPGAs offer attractive decoding efficiency, but are ill suited to sustaining a high-concurrency serving workload alone. This paper presents HyCoInfer, a heterogeneous cloud-edge serving system that assigns prefill to cloud GPUs and distributes decoding across cloud GPUs and edge FPGAs. The design addresses three barriers to practical offloading: capacity balancing across heterogeneous accelerators, the large KV cache crossing a constrained wide-area link, and the latency added by heterogeneous execution. HyCoInfer incorporates FPGA decode capacity into GPU prefill/decode role partitioning and places requests according to queue load, prompt length, and network state. It further combines layer-wise pipelined transfer, decoder-side precomputation, and bandwidth-adaptive KV cache compression based on residual vector quantization. We implement HyCoInfer using two NVIDIA V100 GPUs and an AMD Xilinx Alveo U280 and evaluate it over 100-1000 Mbps links. At 500 Mbps, HyCoInfer delivers up to 4.11 times the request goodput of cloud-only vLLM and reduces mean end-to-end latency by up to 61.1%. Across different network conditions and workloads, the system improves overall serving capacity, with the largest gains on communication-intensive tasks, and reduces per-request energy.
Comments: 10 pages, 13 figures, 3 tables
Subjects: Operating Systems (cs.OS); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22086 [cs.OS]
  (or arXiv:2611.22086v1 [cs.OS] for this version)
  https://doi.org/10.48550/arXiv.2611.22086

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xiangwen Zhuge [view email]
[v1] Tue, 17 Nov 2026 02:14:54 UTC (1,548 KB)

来源:arXiv cs.AI · arxiv.org