arXiv cs.CL· Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun, Liqun Ma, Zhiqiang Shen·· 3 天前AI 评分22
超越规模：梯度如何影响大语言模型的剪枝决策
Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
AI 导读
研究者提出免训练的剪枝方法 GBLM-Pruner，利用预训练 LLM 的归一化梯度（基于 Taylor 展开一阶项）确定剪枝指标，无需后续重训练或权重更新。在 LLaMA 系列、Qwen 系列和 DeepSeek 模型上，该方法在零样本、STEM 推理和代码基准上显著超越 magnitude pruning 与 Wanda，并与需大量权重更新的 SparseGPT 表现相当。
正文 · 原文
Abstract:Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the pruning metric, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguingly, by incorporating gradients, unstructured pruning with our method tends to reveal some structural patterns, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-series, Qwen-Series and Deepseek models across zero-shot, STEM-Reasoning and Coding benchmarks show that GBLM-Pruner surpasses magnitude pruning and Wanda by significant margins and performs competitively against SparseGPT which involves intensive weight updates. We further extend our approach on Vision Transformer and Vision Language Models. Our code and models are available at this https URL.
| Comments: | Code and models at this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2311.04902 [cs.CL] |
| (or arXiv:2311.04902v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2311.04902 arXiv-issued DOI via DataCite |
Submission history
From: Zhiqiang Shen [view email]
[v1]
Wed, 8 Nov 2023 18:59:54 UTC (4,751 KB)
[v2]
Mon, 8 Apr 2024 22:42:49 UTC (3,949 KB)
[v3]
Wed, 14 Oct 2026 08:28:09 UTC (2,567 KB)
来源：arXiv cs.CL · arxiv.org