arXiv cs.CL· Di Wu, Shuaidong Pan·· 13 小时前AI 评分31
GRAM：基于子空间对齐的分组秩感知模型合并方法
GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment
AI 导读
研究者提出分组秩感知模型合并方法 GRAM，利用任务向量之间的主角度几何关系判断哪些任务应合并，再在每组内执行标准 SVD 合并，合并时仅需微调权重。仅增加一个模型（K=2）时，GRAM 显著缓解子空间干扰，在 GLUE（Flan-T5）、8 任务 CLIP 视觉以及 Qwen2-7B-Instruct 的四个中英文医疗 LoRA 指令微调上均优于此前合并方法。
正文 · 原文
Abstract:Merging different models into a single model is highly desirable for multi-task deployment, particularly in regulated domains like healthcare. In such settings, models are often locally fine-tuned due to strict data privacy policies, necessitating that aggregation operates exclusively on the fine-tuned weights. Existing methods apply a uniform merge across all tasks, ignoring the geometric relationships among their learned subspaces. When task updates occupy misaligned subspaces, projecting them onto a shared low-rank basis discards task-relevant directions, a failure mode named \emph{subspace interference}. Through this study, we show that the per-task projection error of a merged model is governed by the principal-angle alignment between each task and the shared subspace, so that grouping geometrically compatible tasks provably reduces this error within each group. Motivated by this observation, we propose \textbf{Group-wise Rank-Aware Model Merging (GRAM)}, which uses the principal-angle geometry of task vectors to decide which tasks should be merged together and then performs a standard SVD-based merge within each coalition; the pipeline requires only the fine-tuned weights at merge time. With only one additional model ($\nK{=}2$), GRAM significantly mitigates subspace interference and consistently outperforms prior merging methods across language understanding (GLUE with Flan-T5), 8-task CLIP vision, and medical instruction tuning (Qwen2-7B-Instruct with four English/Chinese medical LoRAs).
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.29110 [cs.AI] |
| (or arXiv:2610.29110v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29110 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuaidong Pan [view email]
[v1]
Wed, 21 Oct 2026 02:50:46 UTC (11,262 KB)
来源：arXiv cs.CL · arxiv.org