DeepBackground: Metamorphic testing for Deep-Learning-driven image recognition systems accompanied by Background-Relevance

Recently, advances in Deep Learning (DL) have promoted the development of DL-driven image recognition systems in various fields, such as medical treatment, face detection, etc., almost achieving the same level of performance as the human brain. Nevertheless, using DL-driven image recognition systems in these safety-critical domains requires ensuring the accuracy and the stability of these systems. Recent research in this direction mainly focuses on using the image transformations for the overall image to detect the inconsistency of image recognition systems. However, the influence of the image background region ( i . e . , the region of the image other than the target object) on the recognition result of the systems and the robustness evaluation of the systems are not considered. To evaluate the robustness of DL-driven image recognition systems about image background region changes, this paper introduces DeepBackground, a novel metamorphic testing method for DL-driven image recognition systems. First, we define a new metric, termed Background-Relevance (BRC) to assess the influence degree of the image background region on the recognition result of the image recognition systems. DeepBackground defines a series of domain-specific metamorphic relations (MRs) combined with BRC and automatically generates many follow-up test images based on these MRs. Finally, DeepBackground detects the inconsistency of these systems and evaluates their robustness about image background changes according to BRC. Our empirical validation on 3 commercial image recognition services and 6 popular convolutional neural networks (CNNs) models shows that DeepBackground can not only evaluate the robustness of these image recognition systems about image background changes according to BRC, but also can detect their inconsistent behaviors. DeepBackground is capable of automatically generating high-quality test input images to detect the inconsistency of the image recognition systems, and evaluating the robustness of these systems about image background changes according to BRC. • This paper proposes a novel metamorphic testing method for Deep-Learning-driven image recognition systems (DeepBackground). • The approach introduces and formulates a new metric: Background-Relevance (BRC), which can assess the robustness of image recognition systems about background changes. • It also can detect the inconsistency of the image recognition systems. • An empirical study on several image recognition systems shows the feasibility and effectiveness of the approach.