arXiv cs.CL· Ferdinand Kapl, Kaitlin Maile, Stefan Bauer, Johannes von Oswald·· 13 小时前AI 评分27
深度增长与循环结构如何塑造语言模型的上下文学习能力
How Depth Growth and Looping Shape In-Context Learning in Language Models
AI 导读
研究分析了最高 3B 参数的预训练语言模型，发现深度增长与循环复用结构的模型比全深度稠密参考模型更依赖上下文信息，在受控任务中更早习得 ICL，且在权重学习固定关联时保留得更强。注意力头追踪显示深度增长会将早期 induction heads 传播到各层，消融实验表明这类模型在少样本表现和后续 token 预测上更依赖 induction 排序的注意力头。
正文 · 原文
Abstract:Looped and depth-grown language models reuse computations across depth and have attracted interest for their reasoning capabilities. Building on prior suggestive evidence of improved in-context-learning (ICL), we investigate how such reuse favors ICL, including how it shapes the acquisition, retention, and mechanisms of this ability. Our analysis spans pretrained language models up to 3B parameters and controlled tasks. The experiments provide evidence for a shared inductive bias toward learning from context: grown and looped models rely more on information in context and, compared with a full-depth dense reference, acquire ICL earlier in controlled tasks and retain it more strongly when learning fixed associations in the weights is also possible. Tracing attention heads through training reveals that depth growth propagates early induction heads across layers. Ablations show that grown and looped models rely more strongly on induction-ranked heads for both few-shot performance and later-token prediction. Together, these findings provide evidence for a shared inductive bias toward learning from context and suggest that (soft) parameter reuse shapes how this ability develops and which attention mechanisms support it.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.31421 [cs.CL] |
| (or arXiv:2610.31421v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31421 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ferdinand Kapl [view email]
[v1]
Thu, 22 Oct 2026 15:21:22 UTC (1,626 KB)
来源：arXiv cs.CL · arxiv.org