arXiv cs.CL· Tao Jing, Chen Kang, Feifei Zhai, Pengyuan Liu, Dong Yu·· 12 小时前AI 评分31
长短期对比剪枝提升剪枝推理模型的思维链可压缩性
Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning
AI 导读
研究者提出 Contrastive Set-Difference（CSD）方法，通过对比同一问题长、短正确解上的神经元激活，识别并优先剪除冗余推理神经元，同时把偏好数据扩展到剪枝模型恒对或恒错的问题。对 DeepSeek-R1-Distill-Qwen-7B 剪枝 20% 后，该方法在四个数学基准上比 RAC 剪枝加同等压缩少用 8–22% 思考 token，准确率差距在 0.5 分以内。
正文
Abstract:Large reasoning models are costly to serve because they are large and because they generate long chains of thought (CoT). Weight pruning and CoT compression address these two costs separately, but they interfere in that pruning adds redundant reasoning on problems the model still solves and incomplete reasoning on those it no longer solves, and on harder problems makes the model less compressible by subsequent length-preference optimization. We trace this to a blind spot on each side: pruning criteria do not distinguish necessary from redundant reasoning, and preference-based CoT compression receives no training signal on problems a pruned model always or never solves. We propose Contrastive Set-Difference (CSD), which identifies neurons enriched in redundant reasoning by contrasting their activations on long and short correct solutions of the same problem and prunes them preferentially, and we extend preference pairs to the problems a pruned model always or never solves. Pruning 20\% of DeepSeek-R1-Distill-Qwen-7B, the combination uses 8--22\% fewer thinking tokens on four math benchmarks than pruning with reasoning-aware calibration (RAC) followed by the same compression, with accuracy within 0.5 points.
| Comments: | 11 pages. Preliminary version; further experiments and revisions are planned |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34336 [cs.CL] |
| (or arXiv:2610.34336v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34336 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tao Jing [view email]
[v1]
Sun, 25 Oct 2026 14:52:21 UTC (38 KB)
来源：arXiv cs.CL · arxiv.org