arXiv cs.AI· Biswa Sengupta·· 10 小时前AI 评分32
前沿模型组件能叠加吗?一项小语言模型的重复析因实验
Do Frontier Model Ingredients Add Up? A Replicated Factorial Study of Small Language Models
AI 导读
一项重复析因实验在同一小语言模型、相同数据和训练预算下比较五种改动,发现 MoE 层、Muon 优化器和 Gated DeltaNet 混合架构带来的增益最大,额外归一化也有帮助,而参数预算匹配的注意力门控平均增益很小。各改动的增益大体可以叠加,但重复运行仍能观察到改动之间的小幅交互作用,重复实验还揭示出初次统计估计被判为噪声的收益。
正文 · 原文
Abstract:Building a small language model requires choosing which published improvements to combine. We compare five changes in the same model, using the same data and training budget. We test selected combinations and repeat each one with a second random seed. Mixture-of-experts layers, Muon optimisation and a Gated DeltaNet hybrid give the largest gains. Extra normalisation also helps, while an attention gate with a matched parameter budget adds little on average. The gains mostly combine, but small interactions between changes appear again in the repeated runs. Repetition also reveals benefits that an initial statistical estimate treated as noise. These findings apply to our small-model training setting; the expert models also have greater total capacity. The practical lesson is to test promising components together and repeat the comparisons before deciding which ones to keep.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24540 [cs.CL] |
| (or arXiv:2611.24540v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24540 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Biswa Sengupta [view email]
[v1]
Wed, 18 Nov 2026 14:48:52 UTC (38 KB)
来源:arXiv cs.AI · arxiv.org