arXiv cs.CL· Hyojung Han·· 9 小时前AI 评分31
超越位宽:NVIDIA Blackwell 上面向多语言 LLM 推理的部署约束混合精度量化
Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell
AI 导读
研究在 NVIDIA B200 上用 vLLM 对 Qwen3-8B 做低比特量化,验证 144 个融合线性模块的原生打包执行,并测试 INT2/INT3、GPTQ、NVFP4 与混合精度配置。
正文 · 原文
Abstract:Low-bit LLM quantization is useful only if a model preserves service-relevant accuracy, executes through supported GPU kernels, and meets measured memory and latency limits. We study Qwen3-8B on NVIDIA B200 with vLLM, including INT2/INT3, GPTQ, NVFP4, and mixed-precision configurations. We verify native packed execution and precision on all 144 fused linear modules and evaluate Korean, English, numerical, and reasoning quality under preregistered gates. Packed INT2 and INT3 execute, but all seven tested INT2 grids degenerate on generation. Isolated module-restoration effects are strongly non-additive; Korean and reasoning sensitivity rankings disagree, and a promising INT4/INT8 development candidate fails independent final evaluation. A configuration with 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head passes all gates on the untouched FINAL-81 set (5,168 items): 5.917 GiB of weight-loading memory (limit 5.98), Korean teacher-correct retention 0.904 (gate 0.90), and TPOT ratio 0.939 relative to BF16. It improves paired accuracy over NVFP4 W4A4 by +1.4 percentage points (Bonferroni-adjusted interval [+0.2, +2.6]), compared with +3.4 points on the selection set. This comparison changes both activation precision and INT8 protection; exploratory analyses suggest that the activation change contributes substantially, but the isolated allocation effect is not established. Final checkpoints were not retained, and bitwise rebuild reproducibility remains unverified.
| Comments: | 18 pages, 2 figures |
| Subjects: | Information Retrieval (cs.IR); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.21974 [cs.IR] |
| (or arXiv:2611.21974v1 [cs.IR] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21974 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hyo Jung Han [view email]
[v1]
Mon, 16 Nov 2026 22:58:32 UTC (90 KB)
来源:arXiv cs.CL · arxiv.org