Towards cost-effective and robust AI microservice deployment in edge computing environments

As a newly emerged promising computing paradigm, Multi-access Edge Computing (MEC) is capable of energizing massive Internet-of-Things (IoT) devices around us and novel mobile applications, especially the computing-intensive and latency-sensitive ones. Meanwhile, featured by the rapid development of cloud-native technologies in recent years, delivering Artificial-Intelligence (AI) capabilities in a microservice way in the MEC environments comes true nowadays. However, currently MEC systems are still restricted by the limited computing resources and highly dynamic network topology, which leads to high service deployment/maintenance cost. Therefore, how to cost-effectively and robustly deploy edge AI microservices in failure-prone MEC environments has become a hot issue. In this study, we consider an edge AI microservice that can be implemented by composing multiple Deep Neural Networks (DNN) models, in this way, features of different DNN models are aggregated and the deployment cost can be further reduced while fulfilling the Quality-of-Service (QoS) constraint. We propose a Three-Dimension-Dynamic-Programming-based algorithm (TDDP) to yield cost-effective multi-DNN orchestration and load allocation plans. For the robust deployment of the yield orchestration plan, we also develop a robust microservice instance placement algorithm (TLLB) by considering the three levels of load balance including applications, servers, and DNN models. Experiments based on real-world edge environments have demonstrated that the proposed orchestration and placement methods can achieve lower deployment costs and less QoS loss when faced with edge node failures than traditional approaches.