arXiv cs.CL· Kerem Sahin, Sheridan Feucht, Adam Belfki, Jannik Brinkmann, Aaron Mueller, David Bau, Chris Wendler·· 13 小时前AI 评分31
超越复制的上下文学习：抽象式 ICL 的训练时序分析
In-Context Learning Beyond Copying: A Training-Time Analysis of Abstractive ICL
AI 导读
研究者提出 Hapax 训练方案，通过剔除归纳电路可预测 token 的损失贡献来抑制复制训练压力，发现抽象式 ICL 能力仍能涌现，尽管归纳复制大幅减少且 Vanilla 训练中的特征性相变消失。对注意力头的分析显示，Hapax 训练的模型发展出的归纳头更少、更弱。结果表明，归纳电路相关涌现现象与抽象式 ICL 涌现之间的发展关联比此前假设更弱。
正文 · 原文
Abstract:The sudden emergence of induction circuits and the ability to perform in-context copying in LLMs coincides with a sharp drop in training loss over a short span of the training process. This co-occurrence has been cited as evidence that it marks the point at which a wide range of in-context learning (ICL) capabilities begin to emerge. To analyze the training dynamics of few-shot abstractive ICL capabilities (i.e., tasks where the answer is not contained in the input context), we propose Hapax, a training regime that omits the loss contribution of tokens predictable by the induction circuit to control training pressure for copying. Using this intervention, we find that abstractive ICL capabilities can still emerge, despite a substantial reduction in inductive copying and the absence of the characteristic phase changes observed during Vanilla training. Analysis of attention heads shows that Hapax-trained models develop fewer and weaker induction heads. Together, these results suggest that the developmental link between emergent phenomena associated with induction circuits and the emergence of abstractive ICL is weaker than previously hypothesized. More broadly, Hapax provides a controlled framework for analyzing how copying-related training signals shape the development of in-context learning.
| Comments: | EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2511.05743 [cs.CL] |
| (or arXiv:2511.05743v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2511.05743 arXiv-issued DOI via DataCite |
Submission history
From: Kerem Sahin [view email]
[v1]
Fri, 7 Nov 2025 22:11:11 UTC (1,489 KB)
[v2]
Tue, 10 Feb 2026 18:10:20 UTC (2,048 KB)
[v3]
Wed, 21 Oct 2026 20:49:48 UTC (5,303 KB)
来源：arXiv cs.CL · arxiv.org