arXiv cs.CL· Edoardo Vaira, Georgios Chavales, Pierre Nugues·· 11 小时前AI 评分31
不同语料，不同技能？小语言模型的配对研究
Different Corpora, Different Skills? A Paired Study of Small Language Models
AI 导读
研究在 1000 万和 3700 万词预算下对比 1911 版《大英百科全书》与 BabyLM 2026 严格预训练语料，训练教师模型、蒸馏的 5800 万参数 Baby-Llama 学生模型及无蒸馏消融模型。
正文 · 原文
Abstract:Small language models are often trained under tight data budgets, where the choice of pretraining corpus may matter as much as model size or training recipe. We compare two contrasting corpora at matched 10-million- and 37-million-word budgets: the 1911 Encyclopaedia Britannica and the BabyLM 2026 strict pretraining corpus. For each corpus, we train teacher models, distilled 58-million-parameter Baby-Llama students, and no-distillation student ablations, using the same model families, training recipes, and evaluation protocol across corpora. Across BLiMP, BLiMP Supplement, EWoK, Entity Tracking, and COMPS, the corpora produce different skill profiles rather than a single quality ranking. BabyLM 2026 is consistently stronger on linguistic acceptability benchmarks, while Britannica is often stronger on Entity Tracking, a synthetic state-tracking task. The contrast appears across teacher and student models and at both budgets, with the cleanest cross-seed replication in the distilled student comparison. Targeted controls show that the central contrast persists after matching training exposure and at a longer diagnostic context; direct teacher ensembling also does not reproduce the 10M Britannica student's Entity Tracking behavior. Entity Tracking magnitudes nevertheless remain sensitive to training dynamics. For this corpus pair, corpus choice can therefore produce different relative skill profiles at small scale, motivating multi-benchmark evaluation when comparing pretraining data.
| Comments: | Published in the Proceedings of the Second BabyLM Workshop (EMNLP 2026). 13 pages |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.07812 [cs.CL] |
| (or arXiv:2611.07812v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07812 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Proceedings of the Second BabyLM Workshop, pages 413-425, Budapest, Hungary, 2026 |
| Related DOI: | https://doi.org/10.18653/2026.babylm-main.33
DOI(s) linking to related resources |
Submission history
From: Edoardo Vaira [view email]
[v1]
Thu, 5 Nov 2026 10:50:54 UTC (66 KB)
来源：arXiv cs.CL · arxiv.org