arXiv cs.AI· Jinbae Seo, Jungin Park, Hyunsung Jang, Namkoo Ha, Jiyoung Lee, Kwanghoon Sohn·· 10 小时前AI 评分38
COTEX:面向多模态智能体的持续工具空间扩展学习
Continual Tool-Space Extension Learning for Multimodal Agents
AI 导读
COTEX 提出持续工具空间扩展学习方法,以适配器级低成本为已训练的多模态智能体新增专用视觉工具,同时保留原有工具使用能力。该方法定位两组参数方向,分别触发工具调用与决定工具选择,并用正交约束让新工具适配器不影响此前的工具选择。实验在两个代表性智能体和三个专用工具上覆盖十项视觉语言基准,原有工具性能最多下降 0.3 分。
正文 · 原文
Abstract:Multimodal agents largely rely on generic tools such as cropping, search, and code execution, leaving specialist vision models outside their tool space. However, bringing such a model into an already trained agent is not straightforward. The challenge is to learn each new tool efficiently while retaining the ability to use existing ones. This work proposes COTEX, continual tool-space extension learning, that adds new tools at efficient adapter-level cost. COTEX locates two sets of parameter directions; one triggers a tool call and thereby opens a multi-turn reasoning trajectory, while the other determines which tool is chosen. The adapter of a new tool is restricted to the calling directions that leave earlier tool choices unaffected by orthogonal constraint. We further introduce a protocol that converts specialist models' existing task-specific annotations into verified tool-use trajectories. Experiments on two representative agents and three specialist tools show gains on ten vision-language benchmarks, with at most a 0.3-point drop on existing tools.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.23810 [cs.AI] |
| (or arXiv:2611.23810v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23810 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jinbae Seo [view email]
[v1]
Wed, 18 Nov 2026 04:14:02 UTC (12,044 KB)
来源:arXiv cs.AI · arxiv.org