结构感知补偿让 Transformer 降规模更高效：Pythia 与 Qwen3 实验

View PDF HTML (experimental)

Abstract:Model families are trained size by size. Can a pretrained large model instead be converted into a smaller sibling? We characterize the 1.4B->410M conversion in Pythia end to end. Representations align strongly across sizes (ridge R^2=0.84); parameters align weakly. Dense weight projection is destructive; a bit-exact control places the fault in basis mixing, which breaks rotary, per-head, GELU, and LayerNorm structure. Residuals after the best-fit linear operator carry no learnable or transferable signal under shuffle controls, so conversion value lives in initialization. Matched-budget continued pre-training separates two independent levers, least-squares compensation (function lever, best zero-shot) and variance-preserving rescale (dynamics lever, best endpoints), with a placement rule read off the architecture: compensation is well-posed exactly where no normalization sits between cut and read; norm-fronted paths take rescale. At 30M tokens compensation beats the best subcloning variant on a width-reduced pair (84.6+-2.1 vs. 90.7+-3.8, 6/6 seeds) and a held-out depth-reduced pair (109.3 vs. 117.9, 3/3 seeds); the two reach parity at 33x the budget, and at ~5x the donor scale the levers anti-synergize. A second family, Qwen3-1.7B->0.6B (RMSNorm, SwiGLU, grouped-query attention), takes the placement rule unchanged but reverses a 2x2 crossing the re-fit with the activation statistic it reads: on Pythia the re-fit carries the gain and the statistic adds 2.3 perplexity; on Qwen3, with far more heterogeneous MLP unit energies, the statistic carries 66 points and the re-fit is a null on top. Informed selection + compensation + rescale is the best arm in both families, 19x and 23x lower perplexity than from scratch at matched budget (2.94 and 3.13 nats), and beats structured pruning plus distillation. Code, checkpoints, and the frozen eval corpus are released.

Comments:	v5: second architecture family (Qwen3-1.7B-Base -> 0.6B-Base) with the 2x2 selection-statistic result reversing across families; headline pair at six seeds; plain-selection baseline at the 6.9B->1.4B scale pair; CKA calibration; Proposition 1; positioning vs GRAIL/SliceGPT; descriptive arm names; code and checkpoints at the linked repositories. 25 pages, 5 figures, 22 tables
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:	arXiv:2608.02829 [cs.LG]
	(or arXiv:2608.02829v5 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2608.02829 arXiv-issued DOI via DataCite

Submission history

From: Ravi Satya Durga Prasad Yenugula [view email]
[v1] Mon, 3 Aug 2026 19:44:08 UTC (483 KB)
[v2] Mon, 10 Aug 2026 18:36:24 UTC (484 KB)
[v3] Sun, 16 Aug 2026 17:25:23 UTC (484 KB)
[v4] Thu, 24 Sep 2026 04:16:56 UTC (486 KB)
[v5] Tue, 27 Oct 2026 19:59:43 UTC (498 KB)