arXiv cs.CL· Yanshu Li, Ruixiang Tang, Junyi Jessy Li, Elias Stengel-Eskin·· 3 天前AI 评分24
ActICL：面向智能体式多模态上下文学习的后训练方法
ActICL: Post-Training for Agentic Multimodal In-Context Learning
AI 导读
ActICL 通过两阶段后训练让 MLLM 智能体以工具增强推理自主获取 ICL 上下文，在七个基准和三个 MLLM 上平均性能比标准相似度检索提升 11.6 个百分点，比最强先前基线高 4.4 个百分点。其获取的上下文也可供前沿模型使用，GPT-5.6-Sol 进行 4-shot ICL 时性能比标准检索提升 3.5 个百分点。
正文 · 原文
Abstract:Multimodal in-context learning (ICL) is an important capability of multimodal large language models (MLLMs). Its performance is closely tied to the demonstrations it uses, making demonstration retrieval a core problem for ICL. Common methods use fixed scoring methods to retrieve all the demonstrations at once, which may introduce redundancy and noise. Context-aware retrieval methods have emerged to address this problem by allowing incremental retrieval from a growing context. However, none of them lets the MLLM learn how to judge what new demonstrations it needs and then carry out progressive retrieval on its own. To this end, we introduce ActICL, which trains an MLLM agent to acquire and use an ICL context via tool-augmented reasoning. An ActICL agent analyzes its own information gaps during iterative reasoning and tool-use actions. Based on these gaps, it performs targeted retrieval to acquire a set of initial candidates. It can then use the existing context to further filter the candidates and refine the context, or it can iteratively perform a new retrieval. When the agent judges that the context is sufficient for the task, or when the resource limit is reached with the user request met, it sends the acquired context to an answering model, either the agent itself or another MLLM. We use a two-stage post-training procedure that jointly optimizes the agent's acquisition and answering. With an adaptive number of demonstrations, ActICL improves average performance by 11.6 percentage points over standard similarity-based retrieval across seven benchmarks and three MLLMs, and by 4.4 points compared to the strongest prior baseline. The contexts acquired by ActICL agents can also be used effectively by frontier models. When GPT-5.6-Sol performs 4-shot ICL, ActICL's contexts improve its performance by 3.5 points over standard retrieval, averaged over three benchmarks.
| Comments: | Project page: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19475 [cs.CV] |
| (or arXiv:2610.19475v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19475 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yanshu Li [view email]
[v1]
Wed, 14 Oct 2026 02:54:40 UTC (2,614 KB)
来源：arXiv cs.CL · arxiv.org