arXiv cs.CL· Daichi Hotta, Yohei Kobashi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo·· 12 小时前AI 评分31
循环计算能否弥补多模态预训练差距？BabyLM 研究给出边界
Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining
AI 导读
在 100M 词预算下，12 层堆叠重复四次的循环 Transformer 在同等推理算力下 BLiMP 比标准 48 层模型高 5.4 个百分点，但多模态训练使两种架构的 BLiMP 准确率均下降约 3.8 点，循环结构未能缓解。探索性无图像对照实验得分比纯文本基线低 5.7 点，加入视觉前缀平均提升 1.85 点，但三组随机种子下该对比仍不确定。
正文 · 原文
Abstract:Multimodal models have often scored below text-only baselines in the BabyLM Challenge (Klerings et al., 2024; Takmaz et al., 2025). We test whether additional recurrent computation helps close this gap. Recurrence increases computation by reusing layers without adding parameters or training data. Using a single implementation, we compare standard and looped Transformers under a 100M-word budget, with three seeds per main condition. Each image is represented by one feature vector from a frozen DINOv2 encoder, used as a prefix token. Caption words count toward the budget, so multimodal training also changes the language data. In a pre-registered text-only comparison, a 12-layer stack repeated four times exceeds a standard 48-layer Transformer by 5.4 BLiMP percentage points at matched inference compute. However, multimodal training lowers BLiMP accuracy by about 3.8 points in both architectures, with no clear mitigation from recurrence. An exploratory loop-side no-image control, trained with the multimodal language data and batching schedule, scores 5.7 points below the text-only baseline. Adding the visual prefix raises accuracy by 1.85 points on average, but this contrast is uncertain with three seeds. These results suggest that changes in language data and its presentation are a major source of the drop. Diagnostic tests show that the models use visual information; its contribution to grammatical generalization remains uncertain under this single-vector interface.
| Comments: | Accepted to the BabyLM Challenge 2026 (Strict track). 20 pages, 5 figures, 12 tables. This version no longer reports a training-time shuffled-feature control that appeared in the workshop version, because its execution could not be verified (see Limitations) |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.06206 [cs.CL] |
| (or arXiv:2611.06206v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06206 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Daichi Hotta [view email]
[v1]
Wed, 4 Nov 2026 09:21:38 UTC (283 KB)
来源：arXiv cs.CL · arxiv.org