Exploring Communication Efficient Strategies in Federated Learning Systems

Federated learning is a distributed learning paradigm that enables various lightweight devices in distributed networks, such as edge devices, to collaboratively generate a machine learning model over a certain number of iterations using their in-house data. Due to the iterative nature of model training, the FL system requires an excessive number of gradient transfer operations among client devices and the server system for the successful generation of ML models. In certain cases, the FL system requires the transfer of a large number of parameters based on the complexity of the model. In reality, model training occurs on lightweight devices with limited communication bandwidth capabilities. When multiple client devices attempt to transfer multiple parameters simultaneously, it puts a significant load on communication channels, leading to communication bottlenecks. This common scenario in FL systems greatly degrades the model generation capabilities and affects the system's performance. To address the issue of communication delays, various methodologies have been proposed in this study. We will be presenting some common strategies in FL systems to achieve communication efficiency, such as data compression, distillation, sparsification, and more. The study will also discuss various aspects of individual methodologies, including their merits and demerits.