arXiv cs.AI· Tao Li, Yulin Tang, Qi Guo, Xianhao Chen·· 10 小时前AI 评分31
SplitLite：面向拆分学习的低秩残差压缩方法
SplitLite: Low-Rank Residual Compression for Split Learning
AI 导读
SplitLite 是一种通信高效的拆分联邦 LoRA 微调方法，利用相邻 epoch 激活与梯度残差的低有效秩结构，仅传输量化截断 SVD 残差因子。在 GLUE 基准上对多款端侧 LLM 的实验显示，激活上行通信成本最多降低 93.5%，总通信成本最多降低 88.6%，且不损失模型性能。
正文 · 原文
Abstract:Federated fine-tuning of on-device large language models (LLMs) faces a significant computing burden. To overcome this limitation, split learning (SL) has emerged as a promising solution, which offloads the primary training workload to a powerful server. However, SL requires exchanging high-dimensional activations and gradients between clients and the server, resulting in prohibitive communication costs. To overcome this challenge, we propose SplitLite, a communication-efficient split federated LoRA fine-tuning method that exploits the low effective rank structure of consecutive-epoch activation and gradient residuals. Our key finding is that, under low-rank LoRA updates with the pretrained backbone frozen, the activation and gradient residuals of the same data sample between adjacent epochs exhibit low effective ranks, with most of their energy concentrated in a small number of singular directions. By exploiting this property, SplitLite transmits only quantized truncated singular value decomposition (SVD) residual factors, thereby significantly reducing both activation uplink and gradient downlink traffic. Extensive experiments on the GLUE benchmark across a series of advanced on-device LLMs demonstrate that our method reduces activation uplink communication costs by up to 93.5\% and total communication costs by up to 88.6\%, without sacrificing model performance.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.23018 [cs.LG] |
| (or arXiv:2608.23018v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.23018 arXiv-issued DOI via DataCite |
Submission history
From: Tao Li [view email]
[v1]
Mon, 24 Aug 2026 09:20:04 UTC (22,389 KB)
[v2]
Sat, 5 Sep 2026 13:12:30 UTC (1 KB) (withdrawn)
[v3]
Tue, 10 Nov 2026 05:02:06 UTC (21,901 KB)
来源：arXiv cs.AI · arxiv.org