Gradient Calibration for Non-I.I.D. Federated Learning

Federated learning (FL) has yielded impressive results in recent years. However, its effectiveness on non-independently and identically distributed (non-i.i.d) data remains challenging. Existing work aims to address this challenge through client selection strategies or modifications to the client objective function. Despite these efforts, we contend that existing methods are unable to fully leverage client gradients. In this paper, we posit that the challenge of non-i.i.d data stems primarily from conflicting gradients among clients, resulting in deviations in both gradient magnitude and direction. Building on this insight, we propose a simple plug-in, named the Federated Gradient Tailor (FGT), to mitigate the non-i.i.d challenge by identifying and calibrating conflicting gradients before federated aggregation. To evaluate the efficacy of our approach, we conduct a series of experiments on simulated non-i.i.d datasets and demonstrate that the Federated Gradient Tailor significantly improves inference accuracy by 5% and achieves 6 times convergence speedup. Our findings highlight the value of explicitly addressing conflicting gradients in non-i.i.d data, as well as the effectiveness of the proposed Federated Gradient Tailor. This contribution has implications for further enhancing the performance of FL in real-world settings with non-i.i.d data distributions.