GSASG: Global Sparsification With Adaptive Aggregated Stochastic Gradients for Communication-Efficient Federated Learning

This paper addresses the challenge of communication efficiency in federated learning by the proposed algorithm called global sparsification with adaptive aggregated stochastic gradients (GSASG). GSASG leverages the advantages of local sparse communication, global sparsification communication, and adaptive aggregated gradients. More specifically, we devise an efficient global top-k' sparsification operator. By applying this operator to the aggregated gradients obtained from the top-k sparsification, the global model parameter is rarefied to reduce the download transmitted bits from O(dMT) to O(k'MT), where d is the dimension of the gradient, M is the number of workers, T is the total number of epochs and k'≤k