arXiv cs.AI· Ling-Xiang Zhang, Tao Hu, Da-Wei Zhou·· 2 天前AI 评分22
CART：面向多模态持续指令微调的兼容路由方法
CART: Compatible Routing for Multimodal Continual Instruction Tuning
AI 导读
针对多模态持续指令微调中投影器与语言模型非对称适配导致的表征漂移和灾难性遗忘问题，研究者提出兼容路由 CART，为两个模块分别维护任务专属专家并通过加权混合组合，同时考虑专家相关性与跨模块兼容性来联合推断权重。CART 还引入相关性引导的专家合并机制以控制模型规模增长，实验显示其达到 SOTA 性能。
正文 · 原文
Abstract:Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world applications often require them to continually acquire new capabilities from sequential tasks, i.e., Multimodal Continual Instruction Tuning (MCIT). In typical MLLMs, a projector is employed to align visual representations with the language model. However, existing methods primarily protect the language model from forgetting while continuously updating the projector across tasks. Such asymmetric adaptation can cause substantial representation drift in the projector, increasing the mismatch between the projector and language model and consequently exacerbating catastrophic forgetting. To address this issue, we propose Compatible Routing (CART), which explicitly coordinates the adaptation of the projector and language model. Specifically, CART maintains task-specific experts for both modules and combines them through weighted mixtures. CART jointly infers their weights by considering both expert relevance and cross-module compatibility, thereby promoting coordinated adaptation between the two modules. To further control model growth, we introduce relevance-guided expert consolidation, which merges experts exhibiting similar relevance profiles across samples within each module. Extensive experiments demonstrate that CART achieves state-of-the-art performance.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.22137 [cs.AI] |
| (or arXiv:2610.22137v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22137 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ling-Xiang Zhang [view email]
[v1]
Thu, 15 Oct 2026 16:11:22 UTC (486 KB)
来源：arXiv cs.AI · arxiv.org