arXiv cs.CL· Nabeelah Faumi, Deshan Sumanathilaka·· 13 小时前AI 评分31
GPT-2 与 DistilGPT-2 知识蒸馏机制可解释性研究：哪些能力在蒸馏后存活
What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2
AI 导读
一项机制可解释性研究对比 GPT-2 与六层学生模型 DistilGPT-2，发现蒸馏后 IOI 电路完全崩塌（准确率从 94.8% 降至 35.2%，name-mover 保留为零），而 Greater-Than 电路通过将教师第 9 层比较头重映射到学生第 4 层实现 87.7% 任务性能恢复（平均 Pearson r = 0.945）。
正文 · 原文
Abstract:Knowledge distillation compresses large language models into compact students that preserve aggregate metrics, yet the mechanistic consequences of depth-reduction distillation remain largely uncharacterized. This work applies linear CKA, logit lens analysis, attention pattern decomposition, sparse autoencoder feature statistics, and activation patching to compare GPT-2 and its six-layer student DistilGPT-2 across two canonical circuits: the Indirect Object Identification (IOI) circuit and the Greater-Than (GT) circuit. The results reveal a compression-collapse dichotomy: the IOI circuit suffers complete mechanistic collapse (DistilGPT-2: 35.2% accuracy vs. 94.8%, zero name-mover preservation), while the GT circuit survives with 87.7% task performance recovery through active remapping of the teacher's layer-9 comparison head into student layer~4 (mean Pearson r = 0.945). These findings demonstrate that circuit compressibility governs mechanistic survival under distillation, and that aggregate metrics are insufficient diagnostics for circuit integrity.
| Comments: | 5 Figure, 9 Table |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.37157 [cs.CL] |
| (or arXiv:2610.37157v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37157 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Deshan Sumanathilaka Mr [view email]
[v1]
Tue, 27 Oct 2026 12:52:58 UTC (2,725 KB)
来源：arXiv cs.CL · arxiv.org