arXiv cs.CL· Pratik Ramesh, George Stoica, Arun Iyer, Leshem Choshen, Judy Hoffman·· 2 天前AI 评分24
解决跨任务干扰：面向模型合并的专家适配方法 RI
Learning Not To Interfere: Adapting Experts For Model Merging
AI 导读
研究者提出轻量适配框架 Resolving Interference（RI），通过将各专家模型解耦为与其他任务空间功能正交，降低模型合并中的跨任务干扰。RI 仅需无标注辅助数据即可应用，在数据稀缺场景下也能使用，将 SOTA 合并方法性能最高提升 3.8%，对未见域泛化最高提升 2.3%。该方法对辅助输入来源稳健，且对合并超参数调优的敏感度显著更低。
正文 · 原文
Abstract:Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: this https URL
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2603.13467 [cs.LG] |
| (or arXiv:2603.13467v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.13467 arXiv-issued DOI via DataCite |
Submission history
From: Pratik Ramesh [view email]
[v1]
Fri, 13 Mar 2026 17:59:50 UTC (1,097 KB)
[v2]
Wed, 14 Oct 2026 17:59:19 UTC (3,269 KB)
来源：arXiv cs.CL · arxiv.org