arXiv cs.CL· Woosang Lim, Haneul Yoo, Vincent-Daniel Yun, Sungjoo Yoo·· 3 天前AI 评分29
LayerTrim：通过注意力保留的 MLP 裁剪解锁推理语言模型的结构化剪枝
Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
AI 导读
LayerTrim 是一种免训练的结构化剪枝方法，通过识别冗余解码层并选择性裁剪其 MLP 特征，同时保留注意力和解码深度，在不使用梯度或恢复训练的情况下维持长文本推理能力。
正文 · 原文
Abstract:Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose \textit{LayerTrim}, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On \texttt{Qwen3-14B}, LayerTrim achieves 1.24$\times$ faster prefill and 19.7\% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19121 [cs.LG] |
| (or arXiv:2610.19121v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19121 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vincent-Daniel Yun [view email]
[v1]
Tue, 13 Oct 2026 19:55:33 UTC (317 KB)
来源：arXiv cs.CL · arxiv.org