arXiv cs.CL· Ryan Solgi, Parsa Madinei, Jiayi Tian, Rupak Swaminathan, Jing Liu, Nathan Susanj, Zheng Zhang·· 10 天前AI 评分22
PGSVD：面向高效 LLM/VLM 的激活感知帕累托引导低秩压缩
Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
AI 导读
研究者提出 PGSVD，一种零样本低秩压缩流程，通过帕累托引导的秩选择和交替最小二乘实现，改进激活感知压缩。该方法将低秩压缩建模为双目标优化，并证明单一统一容差可产生代理帕累托最优的异构秩。在 LLM 和 VLM 上，PGSVD 在相同压缩率下取得更高精度并带来推理加速。
正文 · 原文
Abstract:Large language models (LLM) and vision-language models (VLM) have achieved state-of-the-art performance, but they impose significant memory and computing challenges in deployment. We present a novel low-rank compression framework to address this challenge. First, we upper bound the change of network loss via layer-wise activation-based compression errors, filling a theoretical gap in the literature. We then formulate low-rank model compression as a bi-objective optimization and prove that a single uniform tolerance yields surrogate Pareto-optimal heterogeneous ranks. Based on our theoretical insights, we propose Pareto-Guided Singular Value Decomposition (PGSVD), a zero-shot pipeline that improves activation-aware compression via Pareto-guided rank selection and alternating least-squares implementation. We apply PGSVD to both LLM and VLM, showing better accuracy at the same compression levels and inference speedup.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.05544 [cs.CL] |
| (or arXiv:2510.05544v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2510.05544 arXiv-issued DOI via DataCite |
Submission history
From: Ryan Solgi [view email]
[v1]
Tue, 7 Oct 2025 03:07:47 UTC (407 KB)
[v2]
Wed, 3 Jun 2026 18:20:52 UTC (408 KB)
[v3]
Tue, 6 Oct 2026 20:51:02 UTC (405 KB)
来源：arXiv cs.CL · arxiv.org