Construction of Semantic Model for GUI of Mobile Applications Using Deep Learning

Modeling the graphical user interface (GUI) of mobile applications is a crucial task for automated robotic testing. Pixel-based modeling methods are non-intrusive and thus have potential for truly black-box automation. However, existing modeling methods can hardly produce accurate models because they do not take the semantic and structural information of GUI into account. In this paper, we propose a layered semantic approach to modeling GUI for mobile applications to address this important problem. The proposed approach adopts a method for recognizing GUI elements and a novel strategy for semantics acquisition using deep learning. The model generated using the proposed approach can support the fully black-box automated robotic testing. We evaluate the approach by conducting a small experiment on 10 mobile applications. The results demonstrate that the proposed approach is effective in generating the GUI models.