跳到正文
arXiv cs.AI· Himanshu Mishra·· 4 小时前AI 评分32

推理重要性归因对语言模型剪枝的脆弱性

The Fragility of Reasoning-Importance Attribution for Pruning Language Models

AI 导读

研究在四个小型推理模型上发现,推理重要性归因测量存在三重脆弱性,且每种都会改变剪枝决策。以最终答案 token 打分会在所有测试的思维链模型中饱和(答案 token NLL 仅 10^-4–10^-3),相比对推理 token 打分,在两个用五个随机种子验证的模型上分别损失 27.6 和 32.2 个准确率点,而困惑度几乎不变、在 7B 上甚至符号反转。

正文 · 原文

View PDF HTML (experimental)

Abstract:Attribution-guided pruning scores components by their causal contribution and protects the important ones, and it is increasingly applied to compress reasoning language models without destroying their chain of thought. Its reliability rests on an unexamined assumption: that reasoning importance can be faithfully measured. Across four small reasoning models, we show this measurement is fragile in three ways, each of which changes pruning decisions. First, scoring final-answer tokens, the natural choice, saturates in every chain-of-thought model we test (answer-token negative log-likelihood, NLL, of $10^{-4}$--$10^{-3}$) because the model states its answer inside its own reasoning; guiding head pruning with this target costs 27.6 and 32.2 accuracy points relative to scoring reasoning tokens on the two models we power with five seeds, while perplexity barely moves and even reverses sign at 7B. Second, causal importance is not localizable to single units: single-neuron effects sit at the measurement floor, their cross-split ranking is noise, and they cancel: a group's ablation effect disagrees in sign with the summed effects of its members in 90\% of groups. Reproducible importance emerges only at groups of $\sim 100$ neurons. Third, contrastively isolated ``reasoning-specific'' importance is not separable from generic importance, and protecting it is worse than protecting raw importance. We additionally pre-registered the hypothesis that distillation-induced concentration of importance predicts the pruning cost; it is not supported, and since our grid confounds concentration with scale, we report it as unresolved rather than reversed. The result is a short set of measurement rules: score reasoning tokens, attribute at coarse granularity, and evaluate with accuracy rather than perplexity.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22234 [cs.AI]
  (or arXiv:2611.22234v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22234

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Himanshu Mishra Mr [view email]
[v1] Tue, 17 Nov 2026 05:36:30 UTC (66 KB)

来源:arXiv cs.AI · arxiv.org