跳到正文
arXiv cs.CL· Arnab Sanyal, Gourav Datta, Sandeep P. Chinchali·· 2 天前AI 评分38

LazyWeights:面向边缘设备的按需解码 LLM 方案

LazyWeights: Decode-on-Demand Edge LLMs

AI 导读

LazyWeights 在保留 GPTQ、AWQ 等部署格式的前提下,通过调优 GPTQ 使整数码可压缩、以 Huffman 编码无损压缩并在 DRAM 与 L2 之间按需解码,让 GPU kernel 无需改动。

正文 · 原文

View PDF HTML (experimental)

Abstract:Memory decides which LLMs an edge device runs, and how fast: capacity on a 4 GB Jetson Nano, bandwidth on an 8 GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, this http URL's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. LazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best this http URL format. A bit-exact 0.21 mm^2 ASIC decoder unit (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, LazyWeights needs 58-64% less time and 53-57% less energy per token than CPU this http URL, and runs a streamed Mistral-7B 6.6x faster at 3.89 bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).
Comments: 6 pages, 1 reference page, 5 figures
Subjects: Machine Learning (cs.LG); Hardware Architecture (cs.AR); Computation and Language (cs.CL); Information Theory (cs.IT)
Cite as: arXiv:2611.17108 [cs.LG]
  (or arXiv:2611.17108v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.17108

arXiv-issued DOI via DataCite

Submission history

From: Arnab Sanyal [view email]
[v1] Thu, 12 Nov 2026 17:03:40 UTC (411 KB)
[v2] Fri, 13 Nov 2026 12:43:24 UTC (411 KB)

来源:arXiv cs.CL · arxiv.org