arXiv cs.AI· Francesco Cagnetta, Casper L. Christensen·· 2 天前AI 评分24
可解主题模型中的上下文学习与专业化：学习理论研究
In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
AI 导读
研究在可解主题模型下为上下文学习与专业化预测器之间的竞争提供了学习理论。理论预测上下文预测器在早期即可用，而专业化预测器需训练 token 数超过数据生成过程的参数量后才出现。训练后的 Transformer 复现了这两个尺度及分离交叉点，且无需拟合参数。
正文 · 原文
Abstract:Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy determines how the model behaves beyond its training distribution, controlling potential capabilities and risks. Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pretraining data vary. However, there is no learning theory for this competition, hence no fundamental understanding of \textit{why} one solution might be favoured over another. Here we provide such a theory in a solvable topic model---documents drawn from a finite mixture of latent topics, mirroring the heterogeneity of modern pretraining data. The dataset comes with two distinct solutions: a specialised predictor, that stores the finite topic mixture, and an in-context one, optimal for topics never seen during training. Our theory predicts their emergence at distinct sample complexities: the in-context predictor is available early, while the specialised one emerges only after the training tokens outnumber the parameters of the data-generating process. Trained transformers reproduce both scales and the separating crossover in remarkable agreement with the theory, with no fitted parameters.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.21901 [cs.LG] |
| (or arXiv:2610.21901v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21901 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Francesco Cagnetta [view email]
[v1]
Thu, 15 Oct 2026 13:54:37 UTC (1,113 KB)
来源：arXiv cs.AI · arxiv.org