跳到正文
arXiv cs.AI· Babak Barazandeh, Subhabrata Majumdar, Vinay Prithyani, George Michailidis·· 4 小时前AI 评分26

Localized LoRA-MoE:自适应路由的低秩专家混合框架

Localized LoRA-MoE: Mixture of Low-Rank Experts With Adaptive Routing

AI 导读

Localized LoRA-MoE 将局部空间分块与上下文条件专家路由结合,提出集中式 Block-Wise 与去中心化 Cell-Wise 两种架构。在 Qwen2.5-1.5B 上,两种路由变体在八个常识基准上均优于静态基线,Cell-Wise 在同等激活容量下比 LoRA 高 3.9 分。

正文 · 原文

View PDF HTML (experimental)

Abstract:Parameter-efficient fine-tuning (PEFT) methods such as LoRA adapt large models through a single low-rank update. Such updates are susceptible to gradient interference: when a training stream interleaves multiple operational contexts, conflicting updates to shared adapter weights collapse the adapter into an unspecialized average. While recent methods introduce block-wise isolation to remedy this issue, their topologies are static and cannot adapt to context switching or localized sensor degradation. In this work, we introduce Localized LoRA-MoE, a framework that combines localized spatial blocking with context-conditioned expert routing. We propose and evaluate two architectural variants: Block-Wise LoRA-MoE (centralized routing), which modulates the entire structural grid through a single context-conditioned gate, and Cell-Wise LoRA-MoE (decentralized routing), which assigns an independent gate to every coordinate cell of the matrix grid. Across three structured transformation benchmarks--a high-dimensional SVD matrix simulation, context-switched transformations over real tabular data, and spatial reconstruction under simulated sensor occlusion--both variants outperform static baselines under comparable adapter budgets. Both variants perform similarly when context shifts the grid uniformly, while cell-wise routing is strictly better when contexts are spatially heterogeneous. When fine-tuning a language model on six interleaved GLUE tasks, task-conditioned routing at a fixed adapter budget outperforms LoRA and static localized adapters, matching per-task adapters with 6x budgets. At decoder scale on Qwen2.5-1.5B, both routed variants outperform static baselines across eight commonsense benchmarks, with cell-wise routing achieving +3.9 points over LoRA at matched active capacity.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2607.05114 [cs.LG]
  (or arXiv:2607.05114v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2607.05114

arXiv-issued DOI via DataCite

Submission history

From: Babak Barazandeh [view email]
[v1] Mon, 6 Jul 2026 14:06:18 UTC (291 KB)
[v2] Tue, 17 Nov 2026 18:47:55 UTC (438 KB)

来源:arXiv cs.AI · arxiv.org