arXiv cs.AI· Yanggan Gu, Shuo Cai, Zihao Wang, Wenjun Wang, Yuanyi Wang, Pengkai Wang, Sirui Huang, Su Lu, Jianmin Wu, Hongxia Yang·· 9 天前AI 评分24
FeatCal：面向模型合并后的特征校准方法
FeatCal: Feature Calibration for Post-Merging Models
AI 导读
FeatCal 通过逐层前向校准合并模型权重来缩小特征漂移，采用闭式解更新权重，无需梯度下降或额外模块。在 CLIP-ViT-B/32 Task Arithmetic 上达 85.5%，超过 Surgery 的 77.0% 和 ProbSurgery 的 78.8%；FLAN-T5-base GLUE 上为 85.2%。
正文 · 原文
Abstract:Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.13030 [cs.LG] |
| (or arXiv:2605.13030v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.13030 arXiv-issued DOI via DataCite |
Submission history
From: Yanggan Gu [view email]
[v1]
Wed, 13 May 2026 05:35:03 UTC (1,128 KB)
[v2]
Thu, 8 Oct 2026 09:28:22 UTC (1,128 KB)
来源：arXiv cs.AI · arxiv.org