arXiv cs.CL· Yaseen M. Osman, Geoff V. Merrett, Stuart E. Middleton·· 12 小时前AI 评分23
基于激活值的主动学习用于上下文学习：挑战与发现
Activation-Based Active Learning for In-Context Learning: Challenges and Insights
AI 导读
研究测试了用模型激活值优化 LLM 上下文示例选择的假设，在 Llama-3.2-3B 和 Qwen2.5-3B 上分析基于 MLP 激活的深度主动学习方法，却发现负结果：MLP 与嵌入层输出（无论从 massive activations 还是前四阶矩视角）与示例质量或任务表现均不相关，所有任务和模型的 Spearman 相关系数绝对值最高仅 0.33。
正文 · 原文
Abstract:Deep active learning has previously been explored for LLM in-context sample selection, but not with methods that utilise recent advances in understanding of transformer activations. In this paper, we test the hypothesis that model activations could provide a fine-grained signal to optimise the selection of in-context examples. We present a comprehensive analysis of MLP activation-based deep active learning methods applied to in-context learning, including how different attention masking strategies impact active learning across diverse classification and generative datasets, using both Llama-3.2-3B and Qwen2.5-3B base models. However, we find a negative result: MLP and embedding layer outputs, viewed through the lenses of massive activations or the first four moments, do not correlate with example quality or task performance. Specifically, the absolute Spearman correlation coefficient is at most 0.33 for all tasks and models we tested, showing that such activation-based sampling should not be used for in-context learning. We hypothesise that this may be due to superposition, whereby models represent more features than they have dimensionality, suggesting that methods like Sparse Autoencoders (SAEs) may be a promising future direction.
| Comments: | Insights workshop at EMNLP 2026. Added auxiliary results |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.05134 [cs.CL] |
| (or arXiv:2606.05134v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.05134 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.18653/2026.insights-1.9
DOI(s) linking to related resources |
Submission history
From: Yaseen Osman [view email]
[v1]
Wed, 3 Jun 2026 17:39:29 UTC (384 KB)
[v2]
Thu, 10 Sep 2026 17:34:58 UTC (805 KB)
[v3]
Sun, 1 Nov 2026 19:09:48 UTC (2,370 KB)
来源：arXiv cs.CL · arxiv.org