arXiv cs.AI· Mohammad Mozaffari, Samuel Kushnir, Maryam Mehri Dehnavi, Amir Yazdanbakhsh·· 12 小时前AI 评分29
OPTIMA：基于二次规划重建的 LLM 一次性剪枝方法
OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction
AI 导读
OPTIMA 是一种一次性训练后剪枝方法，将掩码选择后的逐层权重重建设为共享同一 Hessian 的列级二次规划问题，可在单个加速器上并行求解且无需微调。
正文 · 原文
Abstract:Post-training pruning cuts the inference cost of large language models, yet it faces a trade-off: simple heuristics that zero weights are fast but degrade accuracy, while principled joint optimization methods recover accuracy but are computationally infeasible at modern scale. One-shot methods such as SparseGPT sit in between and apply efficient but approximate weight updates. To close this gap, we introduce OPTIMA, a practical one-shot post-training pruning method that balances accuracy and scalability. OPTIMA casts layer-wise weight reconstruction after mask selection as independent, column-wise quadratic programs (QPs) that share one Hessian per weight matrix. Solving these QPs yields, at solver convergence, the per-column minimizer of the layer-wise reconstruction objective under the shared Hessian, and the shared Hessian makes the QPs easy to batch on accelerators. We implement an accelerator-friendly QP solver that accumulates one Hessian per weight matrix and solves many small QPs in parallel with a convergence certificate per column, enabling one-shot post-training pruning at scale on a single accelerator without fine-tuning. OPTIMA integrates with existing mask selectors and improves average zero-shot accuracy in 40 of 49 (model, mask selector, sparsity) settings across the LLaMA and Gemma families, with mean gains of +0.61, +0.56, and +1.01 points at 50% unstructured, 2:4, and 60% unstructured sparsity, and up to +3.97 points at 60% sparsity, while matching ADMM-based exact reconstruction on its own masks. On an NVIDIA H100, OPTIMA prunes an 8B-parameter transformer end-to-end in 40 hours and 60 GB of peak GPU memory. This shows that principled column-wise QP reconstruction is a practical alternative to heuristic weight updates in one-shot post-training pruning.
| Comments: | Accepted at ACML 2026 (PMLR 353). 32 pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Performance (cs.PF) |
| Cite as: | arXiv:2512.13886 [cs.LG] |
| (or arXiv:2512.13886v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.13886 arXiv-issued DOI via DataCite |
Submission history
From: Mohammad Mozaffari [view email]
[v1]
Mon, 15 Dec 2025 20:41:29 UTC (306 KB)
[v2]
Wed, 31 Dec 2025 02:49:31 UTC (304 KB)
[v3]
Tue, 20 Oct 2026 03:20:26 UTC (291 KB)
来源：arXiv cs.AI · arxiv.org