arXiv cs.CL· Yanggan Gu, Yuanyi Wang, Zhen Li, Shuo Cai, Yuhang Liu, Junzhuo Li, Zihao Wang, Hongxia Yang·· 9 天前AI 评分24
SMAT：简单高效的合并感知训练方法
SMAT: Simple and Efficient Merge-Aware Training
AI 导读
SMAT 将常见模型合并操作归纳为 Scale、Mask、Perturb 三种，通过采样缩放系数、掩码和加性噪声在模拟合并参数上联合优化专家损失与期望损失。在四种语言与视觉语言骨干模型上，SMAT 将五种合并方法的平均得分较最强基线提升 1.07-2.16 分，训练时间开销较标准微调不足 2%。该方法每步仅需一次前向和一次反向传播，并引入周期性调度、算子融合与参数存储切换来提升效率。
正文 · 原文
Abstract:Model merging integrates the capabilities of multiple experts without joint retraining, but standard expert training optimizes task loss alone and does not guarantee good performance after merging. Merge-aware training (MAT) aims to improve merged performance, but existing methods do not fully account for common merging operations and add training cost. We observe that, from an expert's perspective, common merging methods can be described by three operations: Scale reweights its own update, Mask removes selected coordinates, and Perturb adds updates from other experts. Based on this view, we introduce SMAT (Simple MAT), which jointly optimizes expert loss and expected loss at simulated merged parameters generated by sampling scaling coefficients, masks, and additive noise. We further introduce periodic scheduling, kernel fusion, and parameter storage switching to make SMAT efficient, with one forward and one backward pass per step. Across four language and vision-language backbones, SMAT improves the mean score across five merging methods by 1.07-2.16 points over the strongest baseline for each backbone, with less than 2% training-time overhead over standard fine-tuning.
| Comments: | 19 pages, 6 figures, 7 tables. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.33437 [cs.LG] |
| (or arXiv:2609.33437v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.33437 arXiv-issued DOI via DataCite |
Submission history
From: Yanggan Gu [view email]
[v1]
Sun, 27 Sep 2026 10:44:16 UTC (458 KB)
[v2]
Thu, 8 Oct 2026 09:24:07 UTC (458 KB)
来源：arXiv cs.CL · arxiv.org