Implementation of load balancing algorithm based on flink cluster

In real-time computing scenarios, Flink is different from the micro-batch processing idea of Storm and Spark, and truly realizes the stream-batch integration, which is more efficient and fast. However, the default resource scheduling algorithm for heterogeneous clusters leads to local load imbalance and affects task execution time and system throughput. To solve this problem, an improved TTNS algorithm is proposed to improve utilization of cluster resources by classifying task types and dynamically monitoring node resources. Experiments showed that the TTNS scheduling algorithm can reduce the processing time by about 10% on average, increase the system throughput by more than 16% compared with the existing default scheduling algorithm, and make more efficient use of computing resources in clusters.