ISA-PredRNN: An Improved Self-Attention PredRNN Network for Spatiotemporal Predictive Learning

Due to the ability to deal with both temporal and spatial information, spatio-temporal prediction models are used in numerous fields. Currently, the mainstream approach is to use a combination of CNN and RNN for spatio-temporal prediction, in which the PredRNN family of networks is an excellent representative. However, CNNs are only good at dealing with local dependencies and RNNs are poor at capturing long-term dependencies. These factors can affect the accuracy of the prediction results. In order to enhance the predictive power of the model, some improvements are proposed in this paper to overcome the above shortcomings. We introduce the self-attention mechanism and the long-term memory into the ST-LSTM, the basic structural unit of PredRNN-V2, which can handle global and long-term dependencies respectively. To bridge the gap between training and inference, we use the sampling strategy with a combination of Reverse Scheduled Sampling and Scheduled Sampling. As precipitation involves relatively complex spatio-temporal factors, we design the special loss function with weights to better capture extremes. Finally, our new model ISA-PredRNN achieves the best results on the three datasets MovingMNIST, KTH and Radar echo dataset in the comparison experiments.