arXiv cs.CL· Michael A. Lepori, Tal Linzen, Ann Yuan, Katja Filippova·· 2 天前AI 评分38
LLM 难以运用上下文中学到的表征
Language Models Struggle to Use Representations Learned In-Context
AI 导读
一项研究检验 LLM 能否把上下文中学到的表征用于下游任务，发现开源权重模型在下一 token 预测和"自适应世界建模"新任务上都难以调用上下文定义的新语义，即便这些语义已编码在其隐层表征中。对闭源 SOTA 推理模型的测试同样显示，性能最强的 LLM 也无法可靠利用上下文给出的新模式。
正文 · 原文
Abstract:Though large language models (LLMs) have enabled great success across a wide variety of tasks, they still appear to fall short of one of the loftier goals of artificial intelligence research: creating an artificial system that can adapt its behavior to radically new contexts upon deployment. One important step towards this goal is to create systems that can induce rich representations of data that are seen in-context, and then flexibly deploy these representations to accomplish goals. Recently, Park et al. (2024) demonstrated that current LLMs are indeed capable of inducing such representation from context (i.e., in-context representation learning). The present study investigates whether LLMs can use these representations to complete simple downstream tasks.
We first assess whether open-weights LLMs can use in-context representations for next-token prediction, and then probe models using a novel task, adaptive world modeling. In both tasks, we find evidence that open-weights LLMs struggle to deploy representations of novel semantics that are defined in-context, even if they encode these semantics in their latent representations. Furthermore, we assess closed-source, state-of-the-art reasoning models on the adaptive world modeling task, demonstrating that even the most performant LLMs cannot reliably leverage novel patterns presented in-context. Overall, this work seeks to inspire novel methods for encouraging models to not only encode information presented in-context, but to do so in a manner that supports flexible deployment of this information.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.04212 [cs.CL] |
| (or arXiv:2602.04212v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2602.04212 arXiv-issued DOI via DataCite |
Submission history
From: Michael Lepori Jr. [view email]
[v1]
Wed, 4 Feb 2026 04:55:07 UTC (222 KB)
[v2]
Fri, 1 May 2026 03:55:21 UTC (1,198 KB)
[v3]
Thu, 15 Oct 2026 17:06:29 UTC (1,198 KB)
来源：arXiv cs.CL · arxiv.org