Optimizing Data Analysis Tasks Scheduling Based on Resource Utilization Prediction

In order to reduce the average response time of data analysis tasks in a decision support system and maximize the utilization of cluster resources, we proposed the Upsa algorithm for task scheduling, which is based on the prediction of resource utilization. It aims to balance the workload of analytical data processing tasks in the cluster and increase task execution efficiency. In Upsa, the random forest (RF) model is employed to accurately determine the resource utilization rate for the subsequent period to provide a rational basis for task allocation. The Upsa method can ignore a server that is about to become overloaded and assign new duties to the most suitable server. As soon as the server's resource utilization rate returns to normal levels, it will resume receiving tasks. We evaluate the effectiveness of Upsa using the TPC-H benchmark. Compared to the well-known scheduling solution Round Robin, our experimental results show that the Upsa method substantially reduces the average response time.