arXiv cs.CL· Guangzhi Xiong, Zhenghao He, Bohan Liu, Sanchit Sinha, Wenqian Ye, Aidong Zhang·· 10 小时前AI 评分31
用任务算子捕捉上下文学习动态
Capturing In-Context Learning Dynamics with Task Operators
AI 导读
研究者提出 Task Operator(TO),将每个注意力头的输出视为其上下文掩码对应版本的仿射变换,并把该变换作为解析推导的更新回放到注意力输出投影上,从而无需重复处理全部示例即可复现 ICL 效果。
正文 · 原文
Abstract:In-context learning (ICL) enables language models to perform new tasks from demonstrations without weight updates. However, every ICL inference requires processing the full set of examples, resulting in inefficient deployments, and how ICL works mechanistically is not fully understood. Prior work compresses ICL into fixed activation vectors extracted from specific layers or positions, but these input-independent interventions fail on complex tasks where the output depends on fine-grained interactions with the input. By analyzing the ICL forward pass, we show that each attention head's output is an affine transformation of its context-masked counterpart, and that the parameters of this transformation are empirically stable across samples for a given task. Building on this, we introduce Task Operator (TO), which replays this transformation as an analytically derived update to the attention output projection. Across lexical, algorithmic, and reasoning tasks, TO achieves the best overall performance among prior methods and substantially narrows the gap between zero-shot inference and ICL. We further show that the extracted knowledge concentrates in a task-specific sparse circuit across layers and positions, and that averaging operators from disjoint demonstration batches enables effective many-shot scaling without expanding the context window. Our code is available at this https URL.
| Comments: | NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.01054 [cs.CL] |
| (or arXiv:2610.01054v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.01054 arXiv-issued DOI via DataCite |
Submission history
From: Guangzhi Xiong [view email]
[v1]
Thu, 1 Oct 2026 04:50:17 UTC (145 KB)
[v2]
Thu, 19 Nov 2026 18:52:44 UTC (150 KB)
来源:arXiv cs.CL · arxiv.org