arXiv cs.CL· Houcheng Jiang, Mao Zheng, Mingyang Song, Qiyong Zhong, Jie Sun, Tianyu Zhang, Junfeng Fang·· 9 天前AI 评分23
ReCal：面向同策略蒸馏恢复的结构化剪枝校准方法
ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
AI 导读
研究者提出 RECAL（Recovery-Aware Calibration），一种即插即用方法，通过在剪枝前调整校准过程来改善结构化剪枝模型的同策略蒸馏（OPD）恢复效果。
正文 · 原文
Abstract:Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery. Because OPD relies on student-generated trajectories, pruning damage that persists after offline distillation can limit its effectiveness. We propose RECAL, Recovery-Aware Calibration, a simple plug-and-play approach that improves OPD recovery by adjusting calibration before pruning. RECAL uses forward KL between an unpruned teacher and a pruned probe to identify teacher-supported predictions disrupted by pruning, then reweights calibration statistics to guide existing pruning criteria toward preserving these predictions. Across multiple models and pruning methods, RECAL consistently improves mathematical reasoning after OPD, achieving gains of up to 16.7 percentage points on AIME, alongside improvements in most code-generation comparisons. Further analysis shows that RECAL reduces residual damage at heavily affected tokens and establishes performance advantages that persist through recovery. These results demonstrate the value of recovery-aware calibration for improving on-policy distillation recovery of pruned reasoning models.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.11332 [cs.CL] |
| (or arXiv:2610.11332v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11332 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Houcheng Jiang [view email]
[v1]
Thu, 8 Oct 2026 06:28:38 UTC (309 KB)
来源：arXiv cs.CL · arxiv.org