FediLoRA：在模态缺失约束下实用地联邦微调基础模型

View PDF HTML (experimental)

Abstract:Federated Learning with LoRA fine-tuning offers an efficient and privacy-aware solution for institutions to collaboratively leverage their large datasets to train VLLMs. However, participating institutions often possess heterogeneous computational resources, resulting in imbalanced LoRA ranks, which pose a major challenge for effective collaboration. In addition, real-world applications in domains such as healthcare and transportation frequently suffer from missing modalities due to user mistakes or device failures, which significantly degrade global model performance in federated settings. To the best of our knowledge, no prior work has addressed these two challenges simultaneously in federated VLLMs. To tackle these issues, we propose FediLoRA, a lightweight federated LoRA aggregation framework that effectively mitigates the impact of missing modalities in heterogeneous environment. FediLoRA is explicitly motivated by the observation that simple averaging and structured editing can jointly benefit both global and personalized models. Our approach achieves strong performance across multiple general-domain and medical-domain benchmark datasets. Additional experiments on healthcare data further demonstrate that FediLoRA is well-suited for practical, real-world deployment scenarios. Our code is released at this https URL.

Comments:	8 pages, 7 figures
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
ACM classes:	I.2.7; I.2.11
Cite as:	arXiv:2509.06984 [cs.LG]
	(or arXiv:2509.06984v4 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2509.06984 arXiv-issued DOI via DataCite

Submission history

From: Lishan Yang [view email]
[v1] Mon, 1 Sep 2025 10:40:13 UTC (4,613 KB)
[v2] Tue, 23 Sep 2025 04:55:03 UTC (4,613 KB)
[v3] Mon, 18 May 2026 12:48:12 UTC (3,304 KB)
[v4] Sat, 24 Oct 2026 08:24:18 UTC (5,283 KB)