arXiv cs.CL· Dengjie Li, Tiancheng Shen, Yao Zhou, Bernard Ghanem, Yibo Yang, Ming-Hsuan Yang·· 12 小时前AI 评分23
SoCo：面向大语言模型压缩的奇异谱优化框架
Optimizing Singular Spectrum for Large Language Model Compression
AI 导读
研究者提出 SoCo，一种基于奇异值分解的大语言模型压缩框架，用可学习对角矩阵为奇异谱分配重要性分数，并通过三阶段训练从粗压缩逐步细化到稀疏化。SoCo 按稀疏化后的重要性分数自适应剪枝，而非依赖奇异值的固定排序，剩余成分可补偿被剪枝部分的损失。在多个 LLM 和基准上，SoCo 的压缩效果超过现有最优方法。
正文 · 原文
Abstract:Large language models (LLMs) have demonstrated remarkable capabilities, yet prohibitive parameter complexity often hinders their deployment. Existing singular value decomposition (SVD) based compression methods simply deem singular values as importance scores of decomposed components. However, this importance ordered by singular values does not necessarily correlate with the performance of a downstream task. In this work, we introduce SoCo (Singular spectrum optimization for large language model Compression), a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner. Concretely, we employ a learnable diagonal matrix to assign importance scores for singular spectrum and develop a three-stage training process that progressively refines these scores from initial coarse compression to fine-grained sparsification-thereby striking an effective balance between aggressive model compression and performance preservation. Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores, rather than relying on the fixed order of singular values. More importantly, the remaining components with amplified importance scores can compensate for the loss of the pruned ones. Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2502.15092 [cs.CL] |
| (or arXiv:2502.15092v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2502.15092 arXiv-issued DOI via DataCite |
Submission history
From: Tiancheng Shen [view email]
[v1]
Thu, 20 Feb 2025 23:18:39 UTC (1,954 KB)
[v2]
Fri, 16 Oct 2026 17:56:38 UTC (1,950 KB)
来源：arXiv cs.CL · arxiv.org