Genetic Programming Based Hyper Heuristic Approach for Dynamic Workflow Scheduling in the Cloud

Workflow scheduling in the cloud is the process of allocating tasks to limited cloud resources to maximise resource utilization and minimise makespan. This is often achieved by adopting an effective scheduling heuristic. Most existing heuristics rely on a small number of features when making scheduling decisions, ignoring many impacting factors that are important to workflow scheduling. For example, the MINMIN algorithm only considers the size of the tasks when making scheduling decisions. Meanwhile, many existing works focused on scheduling a static set of workflow tasks, neglecting the dynamic nature of cloud computing. In this paper, we introduce a new and more realistic workflow scheduling problem that considers different kinds of workflows, cloud resources, and impacting features. We propose a Dynamic Workflow Scheduling Genetic Programming (DSGP) algorithm to automatically design scheduling heuristics for workflow scheduling to minimise the overall makespan of executing a long sequence of dynamically arriving workflows. Our proposed DSGP algorithm can work consistently well regardless of the size of workflows, the number of available resources, or the pattern of workflows. It is evaluated on a well-known benchmark dataset by using the popular WorkflowSim simulator. Our experiments show that scheduling heuristics designed by DSGP can significantly outperform several manually designed and widely used workflow scheduling heuristics.