arXiv cs.CL· Somin Wadhwa, Silvio Amir, Byron C. Wallace·· 14 小时前AI 评分31
Circuit Distillation：通过对齐内部电路组件蒸馏 Llama3 模型能力
Circuit Distillation
AI 导读
研究者提出 circuit distillation，通过匹配教师与学生模型中功能对应的电路组件并对齐其内部表示，而非仅模仿输出行为。在 Llama3 系列的实体追踪与心智理论（ToM）任务上，该方法优于标准蒸馏，仅调整少量目标参数即成功迁移算法能力。
正文 · 原文
Abstract:Model distillation typically focuses on behavioral mimicry, where a student model is trained to replicate a teacher's output while treating its internal computations as a black box. In this work we propose an alternative approach: Distilling the underlying computational mechanisms implemented by a teacher model. Specifically, we propose circuit distillation, which introduces an objective to align internal representations between analogous circuit components in teacher and student models. We propose a method to match ``functionally correspondent'' circuit components and introduce a loss reflecting similarities between the representations that these induce. We evaluate circuit distillation on entity tracking and theory of mind (ToM) tasks using models from the Llama3 family. Our results demonstrate that circuit distillation outperforms standard distillation, successfully transferring algorithmic capabilities by adjusting only a small, targeted subset of student model parameters. This work establishes the feasibility of transferring mechanisms, which may in turn allow for efficient distillation of targeted teacher capabilities via interpretable and controllable internal student mechanisms.
| Comments: | Preprint; Under Review |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2509.25002 [cs.CL] |
| (or arXiv:2509.25002v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2509.25002 arXiv-issued DOI via DataCite |
Submission history
From: Somin Wadhwa [view email]
[v1]
Mon, 29 Sep 2025 16:28:36 UTC (702 KB)
[v2]
Mon, 26 Oct 2026 14:42:00 UTC (736 KB)
来源：arXiv cs.CL · arxiv.org