arXiv cs.AI· Wei Wang·· 12 小时前AI 评分38
FuTransformer：在 Hopper 上用 FP8 prefill 与 INT4 decode 加速 Llama 式推理
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
AI 导读
FuTransformer 是面向 NVIDIA Hopper GPU 的 Llama 式 Transformer 块 CUDA 实现，prefill 用 FP8、decode 用 INT4 投影权重与 INT4 KV-cache 副本。
正文 · 原文
Abstract:Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache placement, folding the fixed RMSNorm gain into weights, writing FP8 output directly with delayed scaling, and supporting prefill without padding. Decode uses INT4 projection weights and an INT4 KV-cache copy. Weights and keys follow Tensor Core fragment layouts for direct dequantization into MMA operands without full intermediate tiles. A block-diagonal Hadamard transform reduces cache quantization error from outlier channels. On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s. For gate/up projection with SwiGLU at the 8B shape, our BF16 path is 1.13x faster than CODA. Our FP8 path is 2.02-2.18x faster than the evaluated BF16 CODA path. Across 8B and 70B shapes at batches 1-32, INT4 decode is 1.3-2.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point: 2.1-3.3x faster than compiled, CUDA-graphed BF16 and 1.3-6.3x faster than the same block using PyTorch's INT4 projection kernel, which targets a single query row and loses to BF16 at batch 32. It is also 31-51% faster than our FP8 path. Complete Llama-3.1-8B decode takes 2.8 ms per step at batch one, versus 8.1 ms for the compiled transformers implementation. Decoding every WikiText-2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 with round-to-nearest weights or 6.99 with GPTQ in the same format; the rotated INT4 cache adds 0.20. The kernels match a quantized reference computation within 0.001 perplexity.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.23796 [cs.AI] |
| (or arXiv:2610.23796v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23796 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wei Wang [view email]
[v1]
Fri, 16 Oct 2026 17:22:01 UTC (36 KB)
来源：arXiv cs.AI · arxiv.org