Contention-aware Adaptive Model Selection for Machine Vision in Embedded Systems

Real-time machine vision applications running on resource-constrained embedded systems face challenges for maintaining performance. An especially challenging scenario arises when multiple applications execute at the same time, creating contention for the computational resources of the system. This contention results in increase in inference delay of the machine vision applications which can be unacceptable for time-critical tasks. To address this challenge, we propose an adaptive model selection framework to mitigate the impact of system contention and prevent unexpected increase in inference delay by trading off the application accuracy minimally. The framework uses a set of hierarchical deep learning models for image classification. It predicts the inference delays of each model and selects the optimal model for each frame considering the system contention. Compared to a fixed individual model with similar accuracy, our framework improves the performance by significantly reducing the inference delay violations against a practical threshold. We implement our framework on Nvidia Jetson TX2 and show that our approach achieves a gain over the individual model by 27.6% reductions in delay violations.