arXiv cs.AI· Evan Chen, Shiqiang Wang, Dong-Jun Han, Shahryar Zehtabi, Christopher Brinton·· 13 小时前AI 评分29
FA-GC：面向联邦自适应优化的梯度校正方法
Correcting Before Adapting: Gradient Correction for Federated Adaptive Optimization
AI 导读
研究者提出联邦自适应梯度校正方法 FA-GC，在梯度进入 Adam 等自适应优化器的滑动平均之前先做校正，而非像 FA-NT 那样只校正最终更新方向。理论证明在无梯度噪声的理想设定下，FA-GC 的方向仅在全局目标驻点处消失，且对 Adam 族、梯度归一化和 Muon 均给出不含客户端异质性项的收敛界。
正文 · 原文
Abstract:Client-drift compensation methods mitigate the impact of data heterogeneity in federated learning (FL). While drift correction is well studied for stochastic gradient descent (SGD)-based FL, its interaction with adaptive client optimizers remains less understood. Adaptive optimizers such as Adam build each update from running averages of past gradients, so correcting only the final update direction leaves those averages, and the drift they contain, unchanged. We propose Federated Adaptive Gradient Correction ({\tt FA-GC}), which corrects each gradient before it enters these running averages, and compare it with Federated Adaptive Naive Tracking ({\tt FA-NT}), which adds the correction to the final direction. We show that, in an idealized setting without gradient noise, the direction FA-GC follows vanishes only at stationary points of the global objective, whereas the direction FA-NT follows can vanish elsewhere when the optimizer is nonlinear. We also prove convergence of FA-GC for Adam-family methods, gradient normalization, and Muon, which orthogonalizes matrix updates. Unlike our bound for FA-NT, these bounds contain no term for client heterogeneity. On seven image and language tasks, FA-GC with Adam needs the fewest rounds on average to reach the target accuracy, and on the three image tasks it also uses less total communication than uncorrected local Adam and FA-NT. On the QNLI question-entailment task with SOAP, which applies Adam in a rotated basis, and Muon under strong heterogeneity, FA-GC gives the highest accuracy, while FA-NT falls below the uncorrected optimizers.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2502.02727 [cs.LG] |
| (or arXiv:2502.02727v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2502.02727 arXiv-issued DOI via DataCite |
Submission history
From: Evan Chen [view email]
[v1]
Tue, 4 Feb 2025 21:21:30 UTC (705 KB)
[v2]
Sat, 8 Feb 2025 14:01:17 UTC (705 KB)
[v3]
Sun, 18 May 2025 02:48:18 UTC (1,366 KB)
[v4]
Mon, 26 Oct 2026 15:41:52 UTC (4,952 KB)
来源：arXiv cs.AI · arxiv.org