A hybrid-attention-ConvLSTM-based deep learning architecture to extract modal frequencies from limited data using transfer learning

This paper leverages each pixel of a picture acquired from a video camera, in which structural dynamic information is contained, in order to decompose spatiotemporal information from such a non-contact virtual sensor array in the same way as traditional accelerometers to extract structural modal frequencies. Attention-based deep neural network architecture is proposed in this work to better visualize the dynamic properties of structures in the existence of noise with a high resolution. The work combines CNNs and Recurrent Neural Networks (RNNs) to predict modal frequencies of structures from a series of consecutive images. High discriminative features of video frames are firstly extracted using the CNN, and then Conv-Long Short-Term Memory (ConvLSTM) is applied to further process the extracted features to capture the temporal dynamics in videos. The attention mechanisms are embedded in the network to ensure the model learns to focus selectively on those frames containing system dynamics. In particular, the proposed computer vision-based deep learning model takes the video of a vibrating structure as the input and successfully estimates the modal frequencies. Transfer learning is applied to cohere the knowledge learned from publicly available datasets to a much more sophisticated structure and estimate the resonant frequencies. The proposed algorithm optimizes the filter design for video processing in a fully automated way without any human intervention and can generalize and transfer that learned information to more complex structures. The model is trained using publicly available generic baseline data (Dataset A) consisting of several simple beam structures with different material properties and sizes and transferred the learned knowledge to unseen data (Dataset B) consisting of an independent turbine blade. It is concluded that the newly proposed method is more autonomous, accurate, and capable of generalizing the model to a new independent dataset using a transfer learning strategy, and the most advantage of the proposed approach is that the trained deep learning architecture has the capability of estimating the resonant frequencies for independent structures and extending the resonant frequency estimations to higher modes.