arXiv cs.AI· Joseph Turnbull, Jamie Simon·· 10 小时前AI 评分31
无需特征学习也能实现上下文学习：核机器视角下的记忆与泛化
You Don't Need Feature Learning to Do In-Context Learning
AI 导读
研究通过核特征基分解分析旋转不变核机器在 ICL 线性回归任务上的学习过程，发现随着样本量增长，模型依次学习记忆解的高阶分量，泛化解近似为记忆解的低阶截断，因此模型先泛化、后随训练退化为记忆。作者据此提出，ICL 任务上的"泛化"解本质上只是对记忆解欠拟合，而核机器能在该任务上泛化说明特征学习并非 ICL 的必要条件；实验训练的 Transformer 也按核理论预测的高阶统计量记忆任务。
正文 · 原文
Abstract:How should we understand memorization vs. generalization in in-context learning (ICL) tasks, and when does one happen vs. the other? To investigate, we study how rotation-invariant kernel machines learn the ICL linear regression task proposed by Raventos et al. (2023). By decomposing the task in the kernel eigenbasis, we obtain a spectral picture of learning in this ICL setting. This picture reveals that as sample size grows, kernel machines learn successively-higher-degree components of the memorizing solution, which depend upon successively-higher-degree statistics of the training task set. The generalizing solution is approximately a low-degree truncation of the memorizing solution, and so the model first generalizes, then degrades into memorization as training proceeds.
The takeaways are threefold. First, if this case is indicative, learning a "generalizing" solution on an ICL task is really just underfitting the memorizing solution! Second, the fact that kernel machines can generalize on this task means that feature learning is inessential to ICL. Third, transformers empirically trained to do ICL are shown to memorize tasks through the higher-order statistics of the training tasks that the kernel-based theory predicts, suggesting it applies even to models capable of feature learning.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13431 [cs.LG] |
| (or arXiv:2611.13431v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13431 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joseph Turnbull [view email]
[v1]
Tue, 10 Nov 2026 01:16:06 UTC (307 KB)
来源：arXiv cs.AI · arxiv.org