arXiv cs.CL· Arnab Sanyal, Gourav Datta, Sandeep P. Chinchali·· 2 天前AI 评分38
LazyWeights:面向边缘设备的按需解码 LLM 方案
LazyWeights: Decode-on-Demand Edge LLMs
AI 导读
LazyWeights 在保留 GPTQ、AWQ 等部署格式的前提下,通过调优 GPTQ 使整数码可压缩、以 Huffman 编码无损压缩并在 DRAM 与 L2 之间按需解码,让 GPU kernel 无需改动。
正文 · 原文
Abstract:Memory decides which LLMs an edge device runs, and how fast: capacity on a 4 GB Jetson Nano, bandwidth on an 8 GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, this http URL's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. LazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best this http URL format. A bit-exact 0.21 mm^2 ASIC decoder unit (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, LazyWeights needs 58-64% less time and 53-57% less energy per token than CPU this http URL, and runs a streamed Mistral-7B 6.6x faster at 3.89 bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).
| Comments: | 6 pages, 1 reference page, 5 figures |
| Subjects: | Machine Learning (cs.LG); Hardware Architecture (cs.AR); Computation and Language (cs.CL); Information Theory (cs.IT) |
| Cite as: | arXiv:2611.17108 [cs.LG] |
| (or arXiv:2611.17108v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17108 arXiv-issued DOI via DataCite |
Submission history
From: Arnab Sanyal [view email]
[v1]
Thu, 12 Nov 2026 17:03:40 UTC (411 KB)
[v2]
Fri, 13 Nov 2026 12:43:24 UTC (411 KB)
来源:arXiv cs.CL · arxiv.org