跳到正文
arXiv cs.CL· Ziyang Zhang, Yubin Jing, Yuyao Li, Haofan Wang, Yichen Gong·· 2 天前AI 评分38

Concertina:将语言模型折叠进自身,从教师权重直接构建学生模型

Folding Language Models into Themselves

AI 导读

研究者提出 Concertina,将教师模型每 N 个相邻层折叠为一个共享块并执行 N 次,配合低秩修正与逐 token 门控,直接从教师权重构建学生模型而非从头训练。

正文 · 原文

View PDF HTML (experimental)

Abstract:Knowledge distillation usually trains a compact student from scratch, which needs a large amount of data even though the teacher's weights already encode most of what the student has to learn. In this work, we explore building the student directly from the teacher's weights. We propose Concertina, which folds every group of N adjacent teacher layers into one shared block that is executed N times. Each loop step carries a low-rank correction initialized from the layer it replaces, and a per-token gate mixes these corrections, to let every token recover the behavior of the original layers from a single shared block. We further design a distillation data recipe that balances general, mathematical, academic and targeted multiple-choice data, to recover all capabilities of the teacher together rather than one at the expense of another. On Qwen3-4B, a 2.27B Concertina model distilled on 22B tokens retains about 90% of the teacher on commonsense and mathematical reasoning and 84% on average across eight benchmarks. It outperforms same-size dense and layer-sharing baselines, as well as Qwen3-1.7B, which is trained on over a thousand times more data. These results suggest that a compact model need not be retrained at scale, since folding the teacher into itself and distilling it on the right data recovers most of its capabilities. Code is available at this https URL.
Comments: 14 pages, 4 figures, 9 tables. Code: this https URL
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2611.18113 [cs.CL]
  (or arXiv:2611.18113v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.18113

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ziyang Zhang [view email]
[v1] Fri, 13 Nov 2026 12:36:00 UTC (912 KB)

来源:arXiv cs.CL · arxiv.org