arXiv cs.AI· Shiting Wang, Yingjie Zou, Ke Li·· 11 小时前AI 评分30
CUSD：分离共同与独特方向的无数据模型合并方法
Separating Common and Unique Directions for Model Merging
AI 导读
研究者提出无数据预合并方法 Common–Unique Subspace Decomposition（CUSD），将任务向量拆分为跨任务共享的低分散共同分量与高分散的任务主导独特残差，共同分量交由下游算术算子合并、独特残差单独保留。
正文 · 原文
Abstract:Model merging combines multiple fine-tuned models without additional training, and arithmetic methods have emerged as a simple yet effective way to merge models without data. However, existing arithmetic methods treat task vectors in parameter space and apply a single element-wise rule to directions that can differ in mergeability. In a shared directional basis, we find a consistent two-regime structure: a low-dispersion common bulk shared across tasks and a high-dispersion task-dominated tail that carries disproportionate energy. These findings motivate Common--Unique Subspace Decomposition (CUSD), a data-free pre-merging method that separates task vectors into common and unique directional components, merges the common components with a downstream arithmetic operator, and retains unique residuals separately. Experiments across $11$ base models, four model families, and four arithmetic-based merging operators show that CUSD improves 42 of 44 model--operator combinations and achieves up to 22.5% improvement over the corresponding baselines. Our code is publicly available at this https URL
| Comments: | Accepted by NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.04108 [cs.LG] |
| (or arXiv:2611.04108v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.04108 arXiv-issued DOI via DataCite |
Submission history
From: Shiting Wang [view email]
[v1]
Tue, 3 Nov 2026 00:32:11 UTC (1,000 KB)
[v2]
Wed, 4 Nov 2026 10:47:34 UTC (1,000 KB)
来源：arXiv cs.AI · arxiv.org