重新思考高稀疏度 LLM 剪枝的校准：从单一数据源到能力覆盖

View PDF HTML (experimental)

Abstract:Post-training LLM pruning relies on a small calibration set, usually chosen heuristically and judged by WikiText perplexity or by mean accuracy on general-domain commonsense benchmarks. This single-axis view hides capability-specific effects: at 60\% sparsity, strategies separated by $2.85$ points in averaged commonsense accuracy diverge by $51.9$ points in code retention, and across 15 sources no single one preserves all capabilities. We treat calibration as a capability-coverage problem. Balanced, a uniform four-way mixture under a fixed budget, tests the principle on real corpora: it outperforms every evaluated single source, and its margin over C4 grows from $+4.6$ points at 30\% sparsity to $+18.8$ at 60\%. The advantage holds across model scale and a second model family, and transfers to held-out benchmarks disjoint from calibration construction. When aligned corpora are unavailable, IGSP approximates that mixture from the base model and the evaluation taxonomy alone, outperforming prior self-generation methods by $7.23$--$12.79$ points on the LLaMA main suite. On Qwen2.5-7B, IGSP exceeds Balanced on both the main suite and held-out benchmarks. Capability coverage is a first-order design variable for high-sparsity pruning.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2606.03328 [cs.LG]
	(or arXiv:2606.03328v4 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2606.03328 arXiv-issued DOI via DataCite

Submission history

From: Hu Xu [view email]
[v1] Tue, 2 Jun 2026 08:38:14 UTC (363 KB)
[v2] Mon, 8 Jun 2026 13:05:20 UTC (417 KB)
[v3] Thu, 30 Jul 2026 06:41:18 UTC (333 KB)
[v4] Thu, 15 Oct 2026 07:19:20 UTC (4,163 KB)