A multi-branch deep neural network model for failure prognostics based on multimodal data

Non-numerical data, such as images and inspection records, contain information about industrial system degradation, but they are rarely used for failure prognostic tasks given the difficulty of automatic analysis. In this work, we present a novel method for prognostics using multimodal data, i.e. both numerical and non-numerical data. The proposed method is based on the development of a multi-branch Deep Neural Network (DNN), each branch of which is a neural network designed for processing a certain type of data. The method is applied to a case study properly designed to reproduce the problem of prognostics using multimodal data by referring to the operation of steam generators. The results show that it is able to accurately predict future degradation level using multimodal data, outperforming other methods using fewer sources of information.